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Preface 


In  2008,  the  U.S.  Air  Force  and  the  Office  of  the  Secretary  of  Defense  (OSD)  began  a  series 
of  intense  reviews  of  Air  Force  source  selection  policies  and  practices.  As  part  of  this  effort, 
during  the  summer  of  2008,  the  Office  of  the  Assistant  Secretary  of  the  Air  Force  for  Acquisi¬ 
tion  (SAF/AQ)  asked  RAND  Project  AIR  FORCE  (PAF)  to  identify  specific  changes  in  poli¬ 
cies  and  practices  that  could  improve  Air  Force  performance  in  Government  Accountability 
Office  (GAO)  bid  protests. 

This  documented  briefing  reports  the  findings  of  the  resulting  research  project,  “Air 
Force  Source  Selections:  Lessons  Learned  and  Best  Practices,”  which  was  conducted  within 
the  Resource  Management  Program  of  PAP  in  fiscal  year  (PY)  2009.  This  project  studied  the 
Air  Porce’s  recent  experience  with  bid  protests  before  GAO  and  documented  lessons  that  can 
be  learned  from  that  experience.  The  Air  Force  asked  PAF  to  examine  all  source  selections  and 
give  special  attention  to  how  the  Air  Force  conducts  source  selections  in  large  acquisitions. 

PAF  conducted  this  analysis  at  the  request  of  Gen  Donald  Hoffman,  as  former  Military 
Deputy  to  SAF/AQ;  Lt  Gen  Mark  D.  Shackelford,  former  Military  Deputy  to  SAF/AQ;  and 
Roger  S.  Correll,  then-Deputy  Assistant  Secretary  of  the  Air  Force  for  Contracting  (SAF/ 
AQC).  They  asked  PAF  to  identify  specific  changes  that  the  Air  Force  can  make  in  its  source 
selection  policies  and  processes  for  complex  acquisitions  to  reduce  the  rate  of  successful  protests. 

To  do  that,  this  study  used  a  variety  of  analytic  methods,  including  a  review  of  relevant 
government  documents,  interviews  with  relevant  officials  inside  and  outside  the  Air  Force, 
econometric  analysis  of  data  from  administrative  federal  databases,  and  a  detailed  review  of 
the  protests  associated  with  two  Air  Force  programs,  the  Combat  Search  and  Rescue  Recovery 
Vehicle  (CSAR-X)  program  and  the  Aerial  Refueling  Tanker  Aircraft  (KC-X)  program.  The 
study  findings  should  interest  policymakers  and  their  staffs  with  responsibility  for  these  pro¬ 
grams,  for  source  selection  more  generally,  and  for  federal  policies  associated  with  bid  protests 
in  source  selections.  The  econometric  methods  may  also  interest  those  seeking  to  use  admin¬ 
istrative  databases  to  test  hypotheses  about  factors  that  affect  policy  outcomes  that  can  be 
described  with  quantitative  data. 

The  companion  documents  for  this  report  are: 

•  Analysis  of  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections  over 
the  Past  Two  Decades,  Thomas  Light,  Frank  Camm,  Mary  E.  Chenoweth,  Peter  Anthony 
Lewis,  and  Rena  Rudavsky  (TR-883-AF).  This  provides  details  on  the  methods  and  find¬ 
ings  of  our  statistical  analyses  of  patterns  in  Air  Force  experience  with  GAO  bid  protests 
since  1990. 
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•  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and 
Options — Executive  Summary,  Frank  Camm,  Mary  E.  Chenoweth,  John  C.  Graser, 
Thomas  Light,  Mark  A.  Lorell,  and  Susan  K.  Woodward  (MG-1077-AF).  This  summa¬ 
rizes  project  findings  for  an  executive  audience. 


RAND  Project  AIR  FORCE 

RAND  Project  AIR  FORCE  (PAF),  a  division  of  the  RAND  Corporation,  is  the  U.S.  Air 
Force’s  federally  funded  research  and  development  center  for  studies  and  analyses.  PAF  pro¬ 
vides  the  Air  Force  with  independent  analyses  of  policy  alternatives  affecting  the  development, 
employment,  combat  readiness,  and  support  of  current  and  future  air,  space,  and  cyber  forces. 
Research  is  conducted  in  four  programs:  Force  Modernization  and  Employment;  Manpower, 
Personnel,  and  Training;  Resource  Management;  and  Strategy  and  Doctrine. 

Additional  information  about  PAF  is  available  on  our  website: 
http://www.rand.org/paf.html 
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Summary 


When  an  offeror  in  an  Air  Force  source  selection  believes  that  the  Air  Force  has  made  an  error 
that  is  large  enough  to  change  the  outcome  of  the  source  selection,  the  offeror  can  file  a  pro¬ 
test  with  the  Office  of  General  Counsel  of  the  GAO.  Following  review,  if  GAO  agrees  that  a 
significant  error  has  occurred,  it  can  suggest  how  the  Air  Force  should  correct  the  error.  GAO 
cannot  force  the  Air  Force  to  follow  its  recommendation,  but  if  the  Air  Force  does  not,  GAO 
must  report  this  to  Congress.  The  Air  Force  almost  always  follows  GAO  recommendations 
when  GAO  sustains  a  bid  protest. 

GAO  sustained  protests  in  two  recent,  large,  and  highly  visible  Air  Force  acquisitions — 
the  CSAR-X  helicopter  program  in  2007  and  the  KC-X  aircraft  program  in  2008.  Both  pre¬ 
cipitated  broad  criticism  of  how  the  Air  Force  conducts  source  selections.  OSD  temporarily 
suspended  the  Air  Force’s  control  of  the  KC-X  source  selection.  Complications  caused  by  the 
CSAR-X  protest  sustainments  ultimately  helped  lead  OSD  to  cancel  the  program.  The  Air 
Force  and  OSD  began  a  series  of  intense  reviews  of  Air  Force  source  selection  policies  and 
practices.  As  part  of  this  effort,  during  the  summer  of  2008,  the  Office  of  the  Air  Force’s  Ser¬ 
vice  Acquisition  Executive  asked  PAF  to  identify  specific  changes  in  policies  and  practices  that 
could  improve  Air  Force  performance  in  GAO  bid  protests.  The  Air  Force  asked  PAF  to  give 
special  attention  to  how  the  Air  Force  conducts  source  selections  in  large  acquisitions.  This 
documented  briefing  reports  the  findings  of  that  PAF  analysis. 


Corrective  Actions  and  Sustained  Protests  Both  Impose  Costs  on  the  Air 
Force 

Although  broad  criticism  of  the  Air  Force  has  emphasized  recent  protests  sustained  by  GAO, 
they  are  only  part  of  the  picture.  When  a  protest  occurs,  the  Air  Force  can  offer  corrective 
action  to  correct  any  error.  For  example,  it  can  offer  to  reevaluate  proposals  submitted,  to 
reopen  evaluation  and  give  offerors  an  opportunity  to  adjust  their  proposals,  to  change  the 
offerors  included  in  the  source  selection,  to  rewrite  the  request  for  proposal  (RFP)  and  start 
the  source  selection  from  scratch,  or  even  to  change  a  final  source  selection  decision  and  award 
a  contract  to  a  different  winner.  If  a  protester  accepts  such  action,  there  is  no  further  need  for 
GAO  to  review  the  protest.  During  2000-2008,  the  Air  Force  experienced  836  protests — 
about  93  a  year.  It  offered  corrective  action  that  offerors  accepted  in  273,  or  33  percent,  of 
these.  It  ultimately  suffered  sustained  protests  in  only  29,  or  3  percent,  of  these. 

What  can  we  make  of  this  pattern?  Presumably,  the  Air  Force  offers  a  corrective  action 
when  it  believes  that  GAO  will  sustain  a  protest  and  suggest  a  corrective  action  at  least  as  oner¬ 
ous  as  the  one  the  Air  Force  offers.  A  protester  accepts  a  corrective  action  early  when  it  believes 
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that  GAO  will  offer  nothing  better.  That  is,  an  early  corrective  action  is  likely  to  prevail  if  the 
protester  and  Air  Force  have  similar  beliefs  about  how  GAO  would  treat  a  protest.  Under  these 
circumstances,  corrective  actions  accepted  up  front  are  likely,  on  average,  to  impose  costs  com¬ 
parable  to  corrective  actions  suggested  by  GAO  at  the  end  of  a  review. 

That  said,  the  Air  Force  need  not  offer  such  proactive  corrective  action  unless  it  deter¬ 
mines  that  GAO  is  likely  to  sustain  the  protest.  GAO  has  denied  86  percent  of  the  protests 
that  it  reviewed  when  the  Air  Force  did  not  offer  early  corrective  action.  As  a  result,  there  are 
good  reasons  to  believe  the  Air  Force  (1)  will  offer  less  early  corrective  action  than  GAO  would 
require  if  it  ruled  against  the  Air  Force  and  (2)  will  not  offer  early  corrective  action  on  uncer¬ 
tain  cases  where  corrective  action  would  impose  large  costs  on  it,  even  though  going  to  a  GAO 
review  imposes  its  own  administrative  costs  and  delays.  Hence,  although  no  reliable  cost  data 
are  available  to  verify  this,  on  average,  we  believe  that  the  costs  to  the  Air  Force  of  an  observed 
sustained  protest  probably  exceed  those  of  an  observed  early  corrective  action.  Nonetheless, 
because  early  corrective  action  is  so  much  more  common,  both  impose  significant  costs.  The 
Air  Force  leadership  should  track  both. 


Overall  Air  Force  Experience  with  Bid  Protests  Has  Been  Positive 

To  keep  protests  in  perspective,  it  is  useful  to  express  them  in  terms  of  the  total  number  of 
contract  awards  the  Air  Force  makes.  The  number  of  protests  as  a  percentage  of  total  contract 
awards  fell  fairly  steadily  from  about  1.7  percent  in  1995  to  0.5  percent  in  2008.  Sustained 
protests  are  so  unusual  that  they  hardly  register  relative  to  the  total  number  or  the  value  of 
contract  awards.  On  average,  during  2000  to  2008,  GAO  sustained  one  protest  for  every  $20 
billion  the  Air  Force  spent  in  acquisitions.  So  few  sustained  protests  have  occurred  that  it  is 
impossible  to  discern  any  trend  in  them.  On  the  other  hand,  the  Air  Force  has  offered  correc¬ 
tive  actions  in  noticeable  numbers.  Through  the  1990s,  it  offered  corrective  actions  in  about 
0.3  percent  of  contract  awards.  From  2001  to  2008,  the  percentage  fell  fairly  steadily,  ending 
well  under  0.2  percent  of  contract  awards.  All  of  these  trends  point  to  steady  improvement  over 
time,  resulting  in  a  need  to  adjust  fewer  than  0.2  percent  of  the  source  selections  associated 
with  contract  awards  by  2008. 


Significant  Trouble  Will  Persist  in  a  Small  Number  of  Sophisticated  Protests 

All  that  said,  the  Air  Force  continues  to  experience  serious  protests  in  large,  complex  acquisi¬ 
tions  that  present  high  stakes  to  their  participants.  The  CSAR-X  and  KC-X  acquisitions  are 
examples  of  such  acquisitions,  but  they  are  not  entirely  representative  of  the  acquisitions  where 
we  expect  continuing  trouble.  Two  special  characteristics  of  these  acquisitions  made  protests 
and  sustainments  more  likely  than  usual.  (1)  They  both  use  preexisting  designs  with  charac¬ 
teristics  that  are  relatively  well  known  to  all  potential  offerors.  This  has  made  it  difficult  for 
the  Air  Force  to  state  its  source  selection  criteria  clearly  without  foreclosing  more  than  one 
offer  and  so  preventing  a  successful  competition.  Lack  of  clarity  ultimately  provided  one  of  the 
grounds  for  the  sustained  protest  in  the  KC-X  program.  (2)  They  both  involve  foreign  offerors. 
The  Air  Force  never  expressed  any  preferences  for  an  American  source  in  these  acquisitions  so 
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long  as  the  award  complied  with  the  Buy  American  Act,1  but  the  possibility  that  an  Air  Force 
decision  might  push  jobs  outside  the  United  States  significantly  increased  the  political  heat  sur¬ 
rounding  these  acquisitions.  High  political  interest  at  a  minimum  complicated  decisionmak¬ 
ing  by  drawing  more  pointed  external  attention  than  usual  to  these  acquisitions. 

But  these  acquisitions  share  two  other  characteristics  that  we  expect  to  see  repeatedly  in 
a  small  number  of  Air  Force  acquisitions  for  the  future.  (1)  They  are  both  large.  The  net  rev¬ 
enues  associated  with  large  acquisitions  are  likely  to  encourage  any  loser  in  a  source  selection 
to  launch  an  aggressive  protest,  even  if  the  cost  of  that  protest  is  substantial.  (2)  The  winner  of 
either  of  these  source  selections  could  expect  to  dominate  the  global  market  for  similar  systems 
for  the  foreseeable  future.  That  is,  losing  a  source  selection  such  as  one  of  these  has  profound 
strategic  implications  for  a  global  company  that,  again,  are  likely  to  encourage  any  loser  to 
launch  an  aggressive  protest. 

The  experience  of  the  CSAR-X  and  KC-X  protests  also  suggests  that  large  firms  are  learn¬ 
ing  how  to  launch  successful,  sophisticated  protests.  They  and  their  outside  counsel  can  surge 
protest  resources,  imposing  a  heavy  burden  on  the  Air  Force  to  respond  effectively  to  a  surge. 
They  may  see  a  growing  opportunity  in  large,  sophisticated  protests  designed  to  stretch  Air 
Force  capabilities.  As  their  outside  counsel  learns  how  to  do  this,  we  expect  them  to  offer  this 
capability  to  other  companies  with  smaller  stakes  when  they  lose  Air  Force  competitions.  All  of 
these  factors  point  to  the  likelihood  that  the  Air  Force  will  continue  to  face  aggressive  protests 
in  a  small  number  of  large,  complex  acquisitions  that  present  high  stakes  to  the  participants. 


Proactive  Defenses  Are  Available 

The  Air  Force  should  focus  on  changing  source  selection  policies  and  practices  in  ways  that 
help  it  counter  such  sophisticated  protests.  As  it  does  so,  it  should  be  cautious  not  to  make 
changes  that  could  hurt  its  long-term  success  in  dealing  with  the  vast  majority  of  GAO  bid 
protests.  It  should  also  be  cautious  about  focusing  too  much  on  issues  specific  to  the  CSAR-X 
and  KC-X  protests.  Rather,  it  should  seek  proactive  defense  against  sophisticated  protests  in 
large,  complex  acquisitions  that  present  particularly  high  stakes  to  the  participants.  We  see 
opportunities  to  make  changes  in  five  areas.  The  Air  Force  Acquisition  Improvement  Plan 
(AIP)  has  already  begun  to  implement  changes  that  move  the  Air  Force  in  the  direction  we 
suggest  here.  But  several  opportunities  exist  to  refine  the  AIP  or  move  beyond  it. 


Increase  Awareness  of  Finely  Tuned  GAO  Bid  Protest  Rules 

Air  Force  personnel  are  often  puzzled  by  GAO’s  arm’s-length  stance  during  bid  protests.  GAO 
basically  believes  that  Congress  has  given  it  the  authority  and  responsibility  to  protect  the 
integrity  of  the  federal  acquisition  system  by  acting  as  a  counterweight  to  federal  agencies  that 
buy  things,  such  as  the  Air  Force.  Where  the  Air  Force  gives  close  attention  to  the  needs  of  the 
warfighter  as  it  enforces  federal  regulations  within  its  own  acquisition  system,  GAO  focuses 


1  The  Buy  American  Act  (BAA  41  U.S.C.  §  lOa-lOd,  passed  in  1933)  exempts  contractors  in  “qualifying  countries”  from 
the  BAA,  treating  them  the  same  as  U.S.  companies  for  contract  award  purposes.  U.S.  firms  are  treated  (reciprocally)  as 
domestic  contractors  in  procurements  that  occur  in  qualifying  countries. 


xii  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and  Options 


on  the  regulations  themselves  and  deliberately  keeps  its  distance  from  any  substantive  issues  in 
Air  Force  acquisitions  where  protests  arise.  By  the  very  nature  of  its  mission,  GAO  is  likely  to 
see  protests  differently  from  most  Air  Force  personnel  when  protests  occur.  Air  Force  person¬ 
nel  will  be  more  successful  in  bid  protests  the  more  they  understand  about  how  GAO  views 
these  protests. 

In  its  reviews  of  bid  protests,  GAO  sees  itself  as  simply  applying  legislative  language  to 
the  specific  facts  relevant  to  each  new  protest.  Over  the  course  of  time,  however,  GAO  has 
developed  a  history  of  decisions  that  it  draws  on  as  precedents  in  new  decisions.  These  histori¬ 
cal  decisions  provide  an  implicit  set  of  rules  that  can  be  summarized  in  terms  of  three  core 
questions:2 

•  Did  the  agency  follow  the  evaluation  criteria  in  the  RFP? 

•  Were  the  agency  determinations  reasonable  and  properly  documented? 

•  Did  the  agency  violate  any  statute  or  regulation? 

With  appropriate  understanding  of  how  these  rules  apply  in  any  particular  setting,  the  Air 
Force  could  use  them  to  (1)  design  and  execute  source  selections  to  avoid  protests  and  (2)  make 
better  informed  decisions  about  when  to  offer  corrective  action  when  protests  occur  to  avoid 
GAO  review.  Two  of  the  ten  grounds  for  protests  sustained  in  the  CSAR-X  and  KC-X  source 
selections  occurred  because,  despite  close  vetting  by  Air  Force  attorneys,  senior  Air  Force  offi¬ 
cials  misread  the  implicit  rules  GAO  used  to  make  decisions.  The  better  Air  Force  decision¬ 
makers  understand  how  GAO  applies  these  implicit  rules  in  practice,  the  easier  it  should  be  to 
avoid  protest  sustainments. 

Simplify  and  Clarify  Requirements  and  Priorities 

GAO’s  most  common  ground  for  sustaining  a  protest  is  a  mismatch  between  the  criteria  stated 
in  the  RFP  and  the  evaluation  of  these  criteria  later  in  the  source  selection.  This  accounted 
directly  for  three  of  the  ten  grounds  for  the  protests  sustained  in  the  CSAR-X  and  KC-X  source 
selections.  The  simpler  and  the  clearer  the  criteria  and  priorities  among  criteria  presented  in 
the  RFP,  (1)  the  less  ambiguous  these  will  be  when  GAO  reviews  them  and  (2)  the  easier  it 
will  be  for  evaluators  to  execute  and  document  their  evaluation  in  a  way  that  complies  with 
GAO’s  expectations.  Ambiguity  in  the  CSAR-X  and  KC-X  requests  for  proposal  gave  GAO 
large  openings  to  impose  its  own  judgment  about  what  a  “reasonable”  person  would  think  the 
Air  Force  intended  in  its  RFP. 

The  AIP  has  given  this  goal  close  attention  and  is  moving  the  Air  Force  in  the  right  direc¬ 
tion.  The  draft  RFP  for  the  new  round  of  the  KC-X  source  selection  reflects  this  new  guid¬ 
ance.  It  dramatically  reduced  the  number  of  factors  included  in  the  trade  space  for  the  trade¬ 
off  among  requirements.  It  stated  the  relative  importance  of  different  requirements  far  more 
clearly.  And,  in  choosing  which  requirements  to  give  the  most  attention  to  in  the  evaluation,  it 
considered  not  only  which  factors  matter  most  to  the  Air  Force  but  also  which  are  most  likely 
to  discriminate  among  proposals  in  the  new  round  of  the  KC-X  source  selection. 


2  We  thank  James  A.  Hughes,  Jr.,  Air  Force  Deputy  General  Counsel  for  Acquisition  (SAF/GCQ),  for  this  succinct 
summary. 
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Focus  Formal  Cost  Estimates  on  the  Instant  Contract 

The  Air  Force  has  a  strong  interest  in  understanding  the  likely  life  cycle  cost  (LCC)  of  any  new 
system  and  gives  this  close  attention  in  its  general  oversight  of  any  new  program.  But  the  only 
costs  relevant  to  a  source  selection  are  those  that  will  help  the  Air  Force  discriminate  among 
proposals.  So,  for  example,  if  the  Air  Force  expects  the  cost  of  its  own  organic  support  of  a 
new  system  to  be  the  same,  no  matter  which  proposal  it  chooses,  that  cost  is  not  relevant  to 
the  source  selection.  Where  it  does  expect  future  costs  to  differ  among  proposals,  uncertain¬ 
ties  about  the  future  can  create  ambiguities  about  how  the  Air  Force  intends  to  evaluate  future 
costs  or  raise  GAO  objections  that  the  Air  Force’s  treatment  of  future  costs  is  unreasonable. 
Such  concerns  accounted  directly  for  two  of  the  ten  grounds  for  sustained  protests  in  the 
CSAR-X  and  KC-X  source  selections  and  contributed  directly  to  a  third. 

The  AIP  has  responded  by  focusing  formal  cost  estimation  on  the  costs  of  the  “instant 
contract” — the  contract  for  the  deliverables  specifically  identified  in  the  RFP.  Where  it  expects 
future  costs  to  differ  among  proposals,  it  captures  engineering  information  about  the  future 
supportability  of  each  system  offered  and  evaluates  this  information  as  part  of  its  technical 
evaluation.  This  allows  the  Air  Force  to  infer  likely  LCC  implications  of  different  proposals 
without  raising  potentially  problematic  issues  about  how  it  views  the  uncertain  future  environ¬ 
ment  in  which  systems  will  operate.  Naval  Air  Systems  Command  (NAVAIR)  has  used  this 
approach  successfully  in  the  past  and  it  appears  to  be  appropriate  for  the  Air  Force. 

When  taking  this  approach,  the  Air  Force  should  anticipate  the  potential  for  confusion 
about  how  it  defines  cost  in  a  source  selection.  In  particular,  OSD  requires  the  Air  Force  to 
estimate  the  LCC  of  a  new  system  as  part  of  a  Defense  Acquisition  Board  (DAB)  process  that 
runs  in  close  parallel  with  the  source  selection  process  the  Air  Force  uses  to  choose  the  source 
of  that  system.  LCC  includes  forecast  development,  production,  operating  and  support,  and 
disposal  costs.  Application  of  the  approach  prescribed  in  the  AIP  will  lead  the  Air  Force  to  use 
one  cost  estimate  in  the  DAB  process  and  another  narrower  estimate  in  the  source  selection. 
GAO  has  obtained  access  to  such  DAB  cost  estimates  in  recent  source  selections.  The  Air  Force 
should  expect  it  to  again  and,  in  anticipation,  explain  clearly  in  the  RFP  why  the  cost  estimate 
used  in  the  source  selection  differs  from  that  used  in  the  DAB  process. 

Tighten  Discipline  Throughout  the  Source  Selection 

The  nature  of  GAO’s  implicit  rules  means  that  small  errors  during  a  source  selection  can  have 
large  consequences.  Small  errors  directly  produced  three  of  the  ten  grounds  for  protest  in  the 
CSAR-X  and  KC-X  source  selection.  The  Air  Force  could  have  avoided  these  errors  only  by 
maintaining  close  and  tight  control  over  the  whole  source  selection  to  ensure  that  all  parts  of 
the  RFP  were  internally  consistent  and  that  every  one  of  hundreds  of  evaluation  issues  opened 
during  the  source  selection  were  properly  closed  and  disposed  of.  But  the  relatively  inexperi¬ 
enced  personnel  in  the  two  source  selections  attempted  to  complete  the  source  selections  too 
quickly  to  sustain  such  control. 

The  AIP  recognizes  this  problem  and  seeks  to  make  future  source  selection  less  schedule- 
driven  and  more  event-driven.  We  encourage  the  Air  Force  to  take  additional  steps  to  improve 
quality  control  in  the  design  of  the  RFP,  execution  of  the  evaluation  phase,  and  justification 
of  the  final  decision.  Just-in-time  on-the-job  training,  based  on  playing  the  roles  that  relatively 
inexperienced  participants  will  have  in  an  upcoming  source  selection,  can  help  them  anticipate 
the  challenges  they  will  face  and  learn  from  mistakes  in  a  setting  where  the  mistakes  do  not 
impose  real  costs  on  the  Air  Force.  Maintenance  of  technically  skilled  and  experienced  mentor- 
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coaches  through  the  duration  of  the  source  selection  can  back-stop  inexperienced  staff  to  some 
degree,  catching  more  errors  in  time  to  allow  effective  solutions.  And  tools  designed  formally 
to  link  requirements  to  data  requested  and  to  evaluation  methods  applied  can  clarify  the  Air 
Force’s  intent  early  in  a  source  selection  and  facilitate  the  execution  of  evaluation  in  a  way  that 
in  fact  instantiates  the  intent  stated  in  the  RFP.  Although  the  Air  Force  has  such  tools,  the 
CSAR-X  and  KC-X  source  selections  did  not  use  them;  the  second  round  of  the  KC-X  source 
selection  is  using  one. 

Tailor  Quality  Assurance  to  the  Needs  of  a  Less-Experienced  Source  Selection  Workforce 

One  way  to  address  the  problems  of  quality  control  within  a  source  selection  that  are  described 
above  is  to  enhance  external  quality  assurance.  The  AIP  proposes  this  in  the  form  of  more 
effective  external  review  teams.  But  for  the  most  part,  the  Air  Force  applied  this  approach  in 
the  CSAR-X  and  KC-X  source  selection.  Their  high  visibility  led  to  intense  oversight  by  the  Air 
Force  and  OSD,  oversight  that  in  effect  provided  a  test  of  the  external  review  process  proposed 
in  the  AIP.  GAO  sustained  ten  grounds  for  protest  in  these  two  source  selections  despite  this 
intense  external  oversight.  That  is  part  of  the  reason  we  favor  greater  emphasis  on  enhanced 
quality  control  within  the  source  selection;  only  such  close  attention  is  likely  to  identify  the 
errors  that  can  lead  GAO  to  sustain  protests. 

But  enhanced  external  quality  assurance  remains  important.  Our  review  of  the  experi¬ 
ence  of  the  CSAR-X  and  KC-X  source  selections  suggests  that  such  oversight  is  more  likely  to 
be  effective  if  it  displays  the  following  four  characteristics: 

1.  It  is  timely.  It  occurs  early  enough  so  that  the  source  selection  evaluation  team  can  cor¬ 
rect  errors  discovered  in  the  RFP,  the  evaluation  process,  or  the  justification  for  the  final 
decision. 

2.  It  is  well  informed.  External  reviewers  arrive  fully  briefed  on  the  current  status  of  the 
source  selection  and  do  not  have  to  burden  the  source  selection  team  while  getting 
smart  enough  to  provide  useful  oversight. 

3.  It  is  technical.  Reviewers  are  familiar  with  the  nuances  of  the  GAO  bid  protest  process 
and  understand  how  they  play  in  a  practical  setting. 

4.  It  is  hands-on.  It  seeks  the  closest  possible  understanding  of  the  details  of  the  source 
selection  to  help  the  source  selection  team  identify  the  kinds  of  small  errors  discussed 
above. 

A  full-time  oversight  team  within  a  source  selection  is  more  likely  than  an  external  qual¬ 
ity  assurance  team  to  have  all  of  these  characteristics.  But  an  external  team  can  potentially 
bring  a  second  set  of  eyes  as  well  as  capabilities  that  are  too  costly  to  commit  to  a  source  selec¬ 
tion  for  its  full  duration. 
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Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 

work  breakdown  structure 

Weapon  Systems  Acquisition  Reform  Act 


CHAPTER  ONE 


Introduction 


Policy  Context 

•  For  2000-08,  USAF  averaged 

-  93  protests  a  year 

-  3  sustained  protests  a  year 

-  $20  billion  of  spending  per  sustained  protest 

•  GAO  sustained  protests  badly  damaged  two  high- 
profile  programs 

-  Combat  Search  and  Rescue  helicopter  (CSAR-X) 
-Aerial  Refueling  Tanker  aircraft  (KC-X) 

•  How  should  the  USAF  react? 
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Policy  Context 

When  an  offeror  participating  in  a  U.S.  Air  Force  (USAF)  source  selection  believes  the  Air 
Force  has  made  an  error  that  unjustly  “prejudices”  its  chance  of  winning  the  source  selec¬ 
tion,  the  offeror  can  file  a  protest  with  the  Office  of  the  General  Counsel  of  the  Government 
Accountability  Office  (GAO).1  During  the  period  fiscal  year  (FY)  2000  to  FY  2008,  the  Air 
Force  experienced  such  bid  protests  in  an  average  of  93  contract  awards  a  year — claims  that  the 
Air  Force  had  made  errors  in  source  selection  that  were  large  enough  to  change  who  received 
a  contract  award.2  On  average,  GAO  sustained  three  of  these  protests  each  year  and  recom- 


1  This  protest  does  not  go  to  the  side  of  GAO  best  known  for  generating  public  policy  studies  for  Congress.  Rather,  it  goes 
to  the  GAO  Office  of  General  Counsel,  which  is  administratively  separated  from  the  rest  of  GAO  and  operates  under  dif¬ 
ferent  governance  arrangements. 

2  This  number  is  based  on  data  from  the  Protest  and  Congressional  Tracking  System  (PACTS)  database  and  counts  of 
root  B-numbers  for  FY  2000-2008.  GAO  gives  a  unique  B-number  to  each  protest  filing.  One  protest  can  have  several 
B-numbers.  Each  source  selection  has  one  root  B-number — for  example,  B-299145;  each  protest  filing  associated  with  that 
source  selection  has  the  same  root  B-number  and  a  different  suffix — for  example,  .2  in  B-299145. 2.  Hence,  a  count  of  root 
B-numbers  counts  the  number  of  source  selections  in  which  protests  occur.  A  count  of  B-numbers  counts  the  number  of 
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mended  that  the  Air  Force  make  significant  changes  in  how  it  had  conducted  the  source  selec¬ 
tions  associated  with  each  of  these  contract  awards.  The  Air  Force  awarded  about  14,800  new 
contracts  each  year  during  this  period;  GAO  sustained  one  protest  for  every  4,900  contracts 
awarded.* * 3 

What  should  we  make  of  this  record  of  performance?  The  need  to  change  three  source 
selections  a  year  does  not  sound  like  a  serious  problem  in  an  organization  that  buys  as  much 
as  the  Air  Force  does.  Over  that  period  of  time,  it  spent  an  average  of  $59.3  billion  a  year  on 
goods  and  services,  suggesting  a  need  to  change  one  source  selection  for  every  $20  billion  dol¬ 
lars  it  spent.4 

Unfortunately,  a  number  of  the  protests  that  GAO  sustained  during  this  period  of  time 
were  highly  visible  and  caused  significant  disruptions  in  resource  and  operational  planning 
in  the  Air  Force.  The  most  notorious  GAO  sustainments  occurred  in  the  source  selections  to 
recapitalize  the  Air  Force’s  Combat  Search  and  Rescue  Recovery  Vehicle  (CSAR-X)  program 
and  Aerial  Refueling  Tanker  Aircraft  (KC-X)  program  fleets.  Two  sustained  protests  in  the 
CSAR-X  program  in  FY  2007  so  disrupted  Air  Force  planning  that  resulting  delays  ultimately 
helped  lead  to  the  cancellation  of  the  program.5  Eight  sustained  grounds  for  protest  in  the 
KC-X  program  in  2008  have  already  delayed  that  program  by  three  years  and  the  source  selec¬ 
tion  remains  uncompleted.6  For  a  period  of  time,  the  Office  of  the  Secretary  of  Defense  (OSD) 
even  removed  the  Air  Force’s  authority  to  oversee  this  source  selection. 

What  kinds  of  experience  has  the  Air  Force  really  had  with  source  selections  in  recent 
years?  If  serious  problems  exist,  what  are  they?  What  causes  them  and  how  can  the  Air  Force 
correct  these  problems?  To  develop  answers  to  these  questions,  the  Assistant  Secretary  of  the 
Air  Force  for  Acquisitions  (SAF/AQ)  and  the  Deputy  Assistant  Secretary  of  the  Air  Force  for 
Contracting  (SAF/AQC)  sponsored  the  project  underlying  this  documented  briefing,  which 
presents  the  findings  of  the  study  and  the  policy  changes  that  the  Air  Force  can  consider  using 
to  improve  its  performance  with  regard  to  bid  protests  in  the  future. 


protest  filings.  Light  et  al.,  2011,  explains  PACTS  in  more  detail  (Thomas  Light,  Frank  Camm,  Mary  E.  Chenoweth,  Peter 

Lewis,  and  Rena  Rudavsky,  Analysis  of  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections  over  the 
Past  Two  Decades ,  Santa  Monica,  Calif.:  RAND  Corporation,  TR-883-AF,  2012). 

3  These  numbers  come  from  the  source  described  in  the  previous  footnote  and  from  the  Individual  Contract  Action 
Record  (DD350)  and  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG)  databases,  described  in  Light  et  al., 
2010. 

4  Expenditure  numbers  come  from  the  DD350  and  FPDS-NG  databases.  They  are  stated  in  constant  2008  dollars. 

5  For  a  description  of  these  protests  and  GAO’s  justification  for  sustaining  them,  see  U.S.  Government  Accountabil¬ 
ity  Office,  Cost  Assessment  Guide:  Best  Practices  for  Estimating  and  Managing  Program  Costs,  GAO-07-1134SP,  Washing¬ 
ton,  D.C.,  July  2007c;  U.S.  Government  Accountability  Office,  Selected  Recent  GAO  Bid  Protest  Decisions,  Washington, 
D.C.,  updated  August  2007d;  and  U.S.  Government  Accountability  Office,  Decision,  Matter  of  Lockheed  Martin  Systems 
Integration-Owego,  Sikorsky  Aircraft  Company,  Files  B-299145.5,  B-299145.6,  Washington,  D.C.,  August  30,  2007e. 

6  For  a  description  of  these  grounds  for  protests  and  GAO’s  justification  for  sustaining  them,  see  U.S.  Government 
Accountability  Office,  Decision,  Matter  of  The  Boeing  Company,  Files  B-311344,  B-311344.10,  B-311344.ll,  B-311344.3, 
B-311344.4,  B-311344. 6,  B-311344.7,  B-311344.8,  Washington,  D.C.,  June  18,  2008. 
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Analytic  Approach 

*  Review  formal  DoD,  NASA,  USAF,  and  GAO  policy  and 
lessons  learned  on  source  selections 

*  Hold  extensive  discussions  with  Air  Force  personnel 

-  Participants  in  development  of  Acquisition  Improvement 
Plan 

-  Participants  in  CSAR-X,  KC-X  source  selections 

-  Stewards  of  PACTS  database 

*  Hold  general  discussions  with  GAO  officials  responsible  for 
reviewing  bid  protests 

*  Conduct  statistical  and  econometric  analysis  of  data  from 
PACTS,  DD350,  and  FPDS-NG  databases  for  1991-2008. 

*  Develop  findings,  potential  changes  in  USAF  policy 
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Analytic  Approach 

To  inform  our  sponsor  about  Air  Force  experience  with  GAO  bid  protests  and  develop  ideas 
for  potential  changes  in  Air  Force  policy  and  practice,  this  project  undertook  the  following 
analytic  tasks: 

1.  We  reviewed  documents  on  general  Air  Force  source  selection  policies  and 
processes,7  comparable  policies  and  processes  elsewhere  in  the  Department 


7  U.S.  Air  Force,  Source  Selection,  Mandatory  Procedure  MP5315.3,  Washington,  D.C.:  Deputy  Assistant  Secretary  (Con¬ 
tracting)  (Policy  Division)  (SAF/AQCP),  March  2009b;  U.S.  Air  Force,  Acquisition  Improvement  Plan,  Washington,  D.C.: 
SAF/AQ,  May  4,  2009c;  Brett  N.  Kayes  (Capt,  USAF),  “Air  Force  GAO  Protest  Trend  Analysis,”  briefing,  Washington, 
D.C.:  Deputy  Assistant  Secretary  (Contracting)  (Contracting  Operations  Division)  (SAF/AQCK),  updated  September 
19,  2008;  Alexander  R.  Slate,  “Best  Value  Source  Selection:  The  Air  Force  Approach,”  Defense  AT&L,  September-Octo- 
ber  2004,  pp.  52-55;  Edward  C.  Martin,  “Source  Selection  Improvements,”  briefing,  Wright-Patterson  AFB,  Ohio:  ASC 
Acquisition  Center  of  Excellence,  April  10,  2007;  and  Edward  C.  Martin  and  Daniel  C.  Fulmer,  “What’s  New  in  Air  Force 
Source  Selection,”  ASC  Acquisition  Center  of  Excellence,  briefing,  World  Congress,  National  Contract  Management  Asso¬ 
ciation,  Session  #711,  Cincinnati,  Ohio,  April  13-16,  2008. 
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of  Defense  (DoD)8  and  elsewhere  in  the  federal  government.9  We  reviewed 
documents  on  GAO  perspectives  and  trends.10  We  reviewed  the  products  of 
the  Source  Selection  Joint  Analysis  Team  (SSJAT)* 11  and  the  formal  Center 
for  Naval  Analyses  (CNA)  review  of  Air  Force  acquisition.12 

2.  We  interviewed  principals  from  each  of  the  teams  involved  in  the  Air  Force 
Acquisition  Improvement  Plan  (AIP),  focusing  on  issues  we  expected  to 
be  relevant  to  bid  protests  but  also  stretching  to  anticipate  how  potential 
changes  in  the  management  of  the  acquisition  workforce,  the  requirements 
determination  process,  the  organizational  structure  of  the  Air  Force  Materiel 
Command  (AFMC),  and  so  on  might  affect  the  Air  Force’s  ability  to  address 
concerns  about  bid  protests.13  We  focused  our  attention  on  the  substantive 


8  Acquisition  Advisory  Panel,  Report  to  the  Office  of  Federal  Procurement  Policy  and  the  United  States  Congress,  Final  Panel 
Working  Draft,  Washington,  D.C.,  December  2006;  Defense  Information  Services  Agency  (DISA),  Acquisition  Desk- 
book:  Source  Selection,  Washington,  D.C.,  March  2000;  Skip  Hawthorne,  “Risk  Based  Source  Selection,”  briefing,  PEO/ 
SYSCOM  Conference,  Washington,  D.C.:  OUSD(AT&L)  DPAP,  November  7,  2006;  Scott  Ilg,  “Best  Value  Source  Selec¬ 
tion  Trade-Offs,”  briefing,  Defense  Acquisition  University,  Germany,  2006;  U.S.  Army,  Army  Source  Selection  Manual, 
Addendum  AA  to  AFARS,  May  16,  2008;  U.S.  Department  of  Defense,  Addendum  to  the  Defense  Acquisition  Structures  and 
Capabilities  Review,  Pursuant  to  Section  814,  National  Defense  Authorization  Act,  Fiscal  Year  2006,  Washington,  D.C., 
June  2007a;  and  U.S.  Department  of  Defense,  Report  of  the  Defense  Acquisition  Structures  and  Capabilities  Review,  Pursuant 
to  Section  814,  National  Defense  Authorization  Act,  Fiscal  Year  2006,  Washington,  D.C.,  June  2007b. 

9  Doug  Kornreich,  “Bid  Protests:  How  to  Avoid  Them,  and  How  to  Win  Them,”  briefing,  Washington,  D.C.:  HHS,  Office 
of  General  Counsel,  General  Law  Division,  2004. 

10  Michael  R.  Golden,  “Evaluating  Cost  or  Price  in  Competitions:  Challenges  Abound,”  Washington,  D.C.:  U.S.  Govern¬ 
ment  Accountability  Office,  presented  at  DoD  Cost  Analysis  Symposium  42,  Williamsburg,  Va.,  February  2009;  Kate  M. 
Manuel  and  Moshe  Schwartz,  GAO  Bid  Protests:  An  Overview  of  Timeframes  and  Procedures,  Washington,  D.C.:  Congres¬ 
sional  Research  Service,  February  11,  2009;  Moshe  Schwartz  and  Kate  M.  Manuel,  GAO  Bid  Protests:  Trends,  Analysis,  and 
Options  for  Congress,  Washington,  D.C.:  Congressional  Research  Service,  February  2,  2009;  U.S.  Government  Account¬ 
ability  Office,  Bid  Protests  at  GAO:  A  Descriptive  Guide,  8th  ed.,  GAO  06-797SP,  Washington,  D.C.,  2006;  U.S.  Govern¬ 
ment  Accountability  Office,  “Bid  Protest  Regulations,”  n.d.;  U.S.  Government  Accountability  Office,  Report  to  Congress 
on  Bid  Protests  Involving  Defense  Procurements,  B-401197,  Washington,  D.C.,  April  9,  2009;  and  many  GAO  decisions  on 
individual  bid  protests. 

11  U.S.  Department  of  Defense,  “Executive  Report,”  Understanding  the  Problem  Subcommittee,  Source  Selection  Joint 
Analysis  Team,  Washington,  D.C.,  April  24,  2009a;  U.S.  Department  of  Defense,  “Guidance  Joint  Framework,”  briefing, 
Source  Selection  Joint  Analysis  Team,  Washington,  D.C.,  April  24,  2009b;  U.S.  Department  of  Defense,  “Recommenda¬ 
tions  for  Best  Practices,”  Source  Selection  Joint  Analysis  Team,  Washington,  D.C.,  April  24,  2009c;  U.S.  Department  of 
Defense,  “Source  Selection  Policy  and  Guidance  Assessment — Initial  Report:  Executive  Summary,”  Source  Selection  Joint 
Analysis  Team,  Guidance  Subcommittee,  Washington,  D.C.,  April  24,  2009d;  and  U.S.  Department  of  Defense,  “Under¬ 
standing  the  Problem  Subcommittee  Briefing,”  Source  Selection  Joint  Analysis  Team,  Washington,  D.C.,  April  24,  2009e. 

12  Gary  E.  Christie,  Dan  Davis,  and  Gene  Porter,  Air  Force  Acquisition:  Return  to  Excellence,  CNA  Independent  Assess¬ 
ment,  CRM  D0019891.A2/Final,  Alexandria,  Va.,  February  2009. 

13  These  documents  describe  the  conduct  of  the  AIP:  Michael  B.  Donley,  “Strengthening  the  Acquisition  Process,”  memo¬ 
randum,  Secretary  of  the  Air  Force,  Washington,  D.C.,  July  18,  2008a;  Michael  B.  Donley,  “Strengthening  the  Acquisition 
Process  (Your  memo,  dated  2  Sep  08),  memorandum,  SecAF,  Washington,  D.C.,  September  9,  2008b;  Sue  C.,  Payton, 
“Implementation  Plan  for  Strengthening  the  Acquisition  Process,  Memorandum,”  Washington,  D.C.:  SAF/AQ,  September 
15,  2008a;  Sue  C.  Payton,  “Strengthening  the  Acquisition  Process  (SECAF  Memorandum,  July  18,  2008),”  memorandum 
with  attachments,  “Point  Paper  on  KC-X  Source  Selection:  Lessons  Learned,”  “Air  Force  Acquisition  Quick  Look:  Terms 
of  Reference,”  Washington,  D.C.:  SAF/AQ,  September  2,  2008b;  Sue  C.  Payton,  “Strengthening  the  Acquisition  Process 
(Draft  Recommendations),  Memorandum,”  Washington,  D.C.:  SAF/AQ,  September  15,  2008c;  and  Rob  Pollock,  “Weekly 
Assessment  of  Implementation  Plan  Progress  to  Strengthen  the  Acquisition  Process,”  briefing,  Washington,  D.C.:  SAF/ 
ACPO,  December  2,  2008.  The  AIP  yielded  many  useful  products,  including  Kathy  Boockholdt,  “Source  Selection  Lessons 
Learned,”  briefing,  Washington,  D.C.:  SAF/ACE,  December  10,  2008;  U.S.  Air  Force,  “USAF  Annual  Bid  Protest  Update, 
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findings  and  recommendations  that  emerged  from  the  development  of  the 
AIP,  not  on  the  process  the  Air  Force  used  to  develop  the  AIP  or  these  find¬ 
ings  or  recommendations.  We  also  spoke  in  depth  with  other  personnel  in 
Headquarters  Air  Force  who  have  had  long-term  responsibilities  for  source 
selection  policy  and  who  continue  to  help  shape  such  policy. 

3.  We  interviewed  senior  officials  inside  and  outside  the  CSAR-X  and  KC-X 
program  offices  who  were  involved  in  both  of  these  programs  before  and 
after  the  sustained  protests  they  experienced.  We  reviewed  documents  from 
those  source  selections.  We  spoke  with  a  number  of  corporate  officials  associ¬ 
ated  with  these  source  selections.  We  have  made  every  effort  to  ensure  that 
the  material  reported  here  does  not  reveal  any  information  that  is  source- 
selection-sensitive  or  corporate-proprietary. 

4.  We  interviewed  the  principals  responsible  for  managing  and  upgrading  the 
PACTS  database.14  They  provided  a  useful  corporate  memory  about  signifi¬ 
cant  past  experience  with  the  data  system. 

5.  GAO  officials  responsible  for  the  bid  protests  associated  with  the  CSAR-X 
and  KC-X  programs  declined  to  discuss  anything  specific  to  these  protests. 
They  view  both  as  still  being  open  and  active  at  this  time.  Senior  GAO  offi¬ 
cials  openly  discussed  broader  issues,  however,  providing  significant  insight 
into  how  GAO  probably  approached  these  two  protests. 

6.  Using  data  from  PACTS,  DD350,  and  FPDS-NG,  we  constructed  formal 
statistical  cross  tabulations  and  econometric  models  of  the  factors  that 
appear  to  have  affected  protest  patterns  in  the  Air  Force  over  the  period 
FY  1994-2008. 15  In  each  case,  significant  cleaning  efforts  were  required  to 
prepare  data  from  these  sources  for  formal  analysis.  Details  of  this  analysis 
are  documented  separately.16 

7.  Using  the  qualitative  and  quantitative  data  we  collected  and  analyzed  in  the 
efforts  described  above,  we  have  developed  a  set  of  findings  for  the  Air  Force, 
as  well  as  a  number  of  changes  in  policies  and  practices  that  could  help  the 
Air  Force  deal  with  the  new  environment  in  which  we  expect  sophisticated 
bid  protests  to  occur  in  the  future. 


FY09Q1,”  briefing,  Washington,  D.C.:  SAF/ACQ,  January  1,  2009a;  U.S.  Air  Force  Acquisition  Chief  Process  Office, 
United  States  Air  Force  ‘Strengthening  the  Acquisition  Process’  Final  Report,  Report  to  the  Secretary  of  the  Air  Force  on  the 
“Strengthening  the  Acquisition  Process,”  Washington,  D.C.,  2009;  and  David  M.  Van  Buren,  “Acquisition  Improvement 
Plan  Implementation,”  Memorandum,  SAF/AQ,  Washington,  D.C.,  May  8,  2009. 

14  Light  et  al.,  2011,  describes  the  PACTS  database. 

11  Light  et  al.,  2011,  briefly  describes  these  databases. 

16  Light  etal.,  2011. 
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Primary  Findings 

•  Corrective  actions  and  sustained  protests  both  impose  costs 
on  USAF.  Track  both. 

•  Overall  USAF  experience  with  bid  protests  has  been  positive. 

•  Expect  trouble  from  sophisticated  protests  in  selected 
circumstances. 

•  Proactive  defenses  are  available: 

-  Increase  awareness  of  finely  tuned  GAO  rules. 

-  Simplify  and  clarify  requirements  and  priorities. 

-  Tighten  discipline  in  RFP,  evaluation,  justification  of  final 
decision. 

-  Address  future  costs  in  mission  capability  factor; 
expect  to  reconcile  with  DAB  cost  estimates. 

-  Tailor  quality  assurance  to  needs  of  a  less  experienced 
source  selection  workforce. 

-  Allow  quality  to  override  schedule  when  unexpected 
problems  arise. 
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Summary  of  Primary  Findings 

Both  GAO  and  the  Air  Force  have  traditionally  tracked  Air  Force  performance  with  regard  to 
bid  protests  by  focusing  on  the  number  of  protests  sustained  by  GAO.  Data  analysis  suggests 
that  the  costs  that  the  Air  Force  experiences  when  it  voluntarily  agrees  to  take  a  corrective 
action  to  reduce  an  error  in  a  source  selection  are  also  important.  As  a  result,  the  Air  Force 
should  track  both  of  these  to  understand  its  own  performance  with  regard  to  bid  protests. 

When  we  track  Air  Force  performance  over  time,  we  find  a  long-term  reduction  in  the 
number  of  protests  per  contract  award  since  FY  1994  and  a  long-term  reduction  in  the  number 
of  corrective  actions  per  contract  award  since  FY  2001.  No  trend  can  be  discerned  in  the 
number  of  sustained  protests,  because  so  few  occurred  over  this  period.  From  FY  1994  to 
FY  2008,  the  Air  Force  had  to  correct  errors — through  proactive  corrective  actions  or  responses 
to  sustained  protests — in  only  0.2  percent  of  the  acquisitions  in  which  it  awarded  contracts; 
that  performance  has  improved  since  FY  2001.  These  observations  paint  a  positive  picture  of 
the  Air  Force’s  broad  experience  with  bid  protests. 

The  recent  bid  protests  experienced  in  its  CSAR-X  and  KC-X  programs  are  not  represen¬ 
tative  of  the  Air  Force’s  broader  experience  and  are  not  likely  to  help  the  Air  Force  predict  its 
general  experience  in  the  future.  But  they  are  characteristic  of  a  specific  form  of  troublesome 
protest  that  the  Air  Force  is  likely  to  see  more  of  as  time  passes,  if  only  in  very  small  numbers. 
These  are  sophisticated  protests  undertaken  by  losers  in  Air  Force  competitions  with  a  great 
deal  at  stake  and  no  business  choice  but  to  search  aggressively  for  errors  in  any  Air  Force  source 
selection  in  which  they  lose.  Such  protesters  have  been  exceptionally  successful  and  will  con- 
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tinue  to  be  unless  the  Air  Force  changes  its  source  selection  policies  and  practices  in  ways  that 
are  explicitly  designed  to  counter  these  sophisticated  efforts. 

The  Air  Force  appears  to  have  many  options  for  dealing  with  protests  of  this  kind.  It  is 
already  trying  some  of  these  in  the  new  KC-X  source  selection.  Options  to  consider: 

•  Continue  to  anticipate  that  awards  in  major  procurements  are  likely  to  be  subject  to  pro¬ 
test.  With  that  in  mind,  sustain  GAO’s  standards  from  end  to  end  in  a  source  selection. 
Ensure  that  the  source  selection  advisory  council  (SSAC)  and  source  selection  authority 
(SSA)  understand  what  GAO  expects  when  decisions  move  to  them. 

•  Air  Force  attorneys  understand  GAO’s  standards  better  than  anyone  else  in  the  Air  Force. 
Take  full  advantage  of  their  knowledge.  Assign  Air  Force  legal  advisors  to  the  source 
selection  evaluation  team  (SSET),  SSAC,  and  the  SSA.  Ensure  that  the  legal  team  sup¬ 
porting  the  procurement  includes  one  or  more  litigation  attorneys  who  are  familiar  with 
the  bid  protest  process. 

•  Simplify  requirements.  Focus  on  things  likely  to  discriminate  among  sources.  Clearly 
define  discriminators  and  their  relative  importance. 

•  Provide  training  for  the  SSET  at  the  beginning  of  the  process.  Emphasize  avoidance 
of  common  mistakes.  For  example,  state  in  the  request  for  proposal  (RFP)  where  the 
Air  Force  will  use  each  piece  of  information  requested.  Follow  the  RFP  to  the  letter  in 
evaluation. 

•  Track  each  evaluation  notice  (EN)  in  real  time  and  dispose  of  it  with  a  clear  justification. 
Follow  through  on  open  evaluation  items,  including  extraneous  material  in  the  source 
selection  record. 

•  Adequately  document  evaluations.  Have  a  plan  to  develop  an  accurate  agency  record  that 
will  be  complete  when  the  SSA  makes  a  decision. 

•  Define  how  cost  calculations  will  be  made  and  used  precisely  in  the  RFP.  Link  elements 
measured  clearly  to  the  evaluation  criteria  used  in  the  source  selection.  If  GAO  detects 
uncertainty  or  questions  about  the  validity  of  an  approach,  it  can  impose  its  own  approach. 

•  Use  an  approach  to  quality  assurance  (QA)  that  improves  on  past  practice  in  four  ways: 

-  QA  specialists  engage  early  enough  to  allow  time  for  the  source  selection  to  correct  any 
errors  they  detect. 

-  They  come  to  a  source  selection  fully  informed  about  its  status  before  they  arrive. 

-  They  come  with  technical  expertise  about  specific  issues  relevant  to  the  source  selection. 

-  They  come  to  get  their  hands  dirty  and  apply  a  hard  wire  brush  to  details  of  the  source 
selection.17 

•  Treat  schedules  during  source  selections  as  goals,  but  ensure  that  everyone  understands 
that  quality  must  override  schedule  when  there  is  a  conflict  between  them. 


17  That  is,  they  should  get  into  the  details  of  whatever  source  selection  process  they  are  examining.  They  should  get  close 
enough  so  that  their  statements  inform  the  execution  of  detailed  tasks  in  the  process.  They  should  provide  immediate  advice 
at  that  level  and  use  the  knowledge  they  gain  from  such  close  observation  to  inform  their  assessment  of  the  performance  of 
the  process  in  broader  terms.  In  this  setting,  the  devil  is  in  the  details,  and  they  must  get  to  know  the  devil. 
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Roadmap  for  the  Document 

The  document  continues  in  Chapter  Two  by  describing  the  processes  that  the  Air  Force  uses 
to  manage  its  source  selections  and  that  GAO  uses  to  manage  bid  protests.  Chapter  Three 
describes  the  findings  of  statistical  analysis  of  broad  patterns  in  the  Air  Force’s  historical  expe¬ 
rience  with  bid  protests.  Chapter  Four  describes  the  sophisticated  protests  sustained  in  the 
CSAR-X  and  KC-X  programs  and  the  lessons  the  Air  Force  can  learn  from  these  to  combat 
future  sophisticated  protests.  Chapter  Five  summarizes  findings  based  on  the  discussions  in 
Chapters  Three  and  Four,  reviews  of  official  documents,  and  discussions  with  knowledgeable 
personnel.  Chapter  Six  closes  the  document  with  a  summary  of  the  changes  that  the  Air  Force 
might  consider  to  improve  its  performance  with  regard  to  bid  protests.  A  separate  technical 
report  describes  the  quantitative  data  sources  used  in  this  document  and  the  formal  statistical 
methods  that  underlie  the  findings  reported  here.18 


18  Light  et  al.,  2011. 
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This  chapter  describes  the  two  processes  relevant  to  the  analysis  in  this  document:  the  process 
that  the  Air  Force  uses  to  manage  a  source  selection  and  the  process  that  GAO  uses  to  manage 
a  bid  protest.  Presenting  the  steps  in  a  bid  protest  after  the  steps  in  a  source  selection  process 
might  suggest  that  a  bid  protest  can  occur  only  after  the  Air  Force  has  chosen  and  announced  a 
winner  of  the  source  selection.  That  is  not  correct.  GAO  has  very  specific  rules  about  timeliness 
that  define  how  soon  an  offeror  must  file  a  protest  after  it  believes  it  has  detected  a  government 
error.  As  a  result,  protests  can  occur  well  before  the  Air  Force  announces  a  winner.1  And  so 
the  series  of  steps  we  associate  with  a  bid  protest  could  begin  quite  early  in  a  source  selection. 


1  For  example,  an  offeror  can  protest  one  of  the  selection  criteria  stated  in  an  RFP. 
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Steps  of  the  Source  Selection  Process 

A  source  selection  typically  includes  the  following  basic  steps:2 

•  development  and  approval  of  requirements 

•  development  and  approval  of  a  source  selection  plan 

•  development,  approval,  and  announcement  of  an  RFP 

•  evaluation  of  proposals 

•  selection,  approval,  and  announcement  of  a  winner. 

Development  and  Approval  of  Requirements 

The  processes  that  the  Air  Force  uses  to  develop  the  requirements  relevant  to  a  source  selection 
are  separate  from  the  source  selection  per  se.  So  this  step  is  not  technically  part  of  source  selec¬ 
tion.  We  include  it  here  because  it  is  so  important  to  the  problems  in  source  selection  that  can 
lead  to  bid  protests. 

Three  requirements  documents  are  particularly  important  to  source  selections  for  large 
acquisitions.  The  first  is  the  Analysis  of  Alternatives  (AoA),  which  is  a 

documented  analytical  evaluation  of  the  performance,  operational  effectiveness,  opera¬ 
tional  suitability,  and  estimated  costs  of  alternative  systems  to  meet  a  mission  capabil¬ 
ity  that  has  been  identified  through  the  Joint  Capabilities  Integration  and  Development 


2  This  description  draws  heavily  on  Slate,  2004;  Martin,  2007;  U.S.  Air  Force,  2009b;  and  U.S.  Department  of  Defense. 
2009b  and  2009c. 
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System  (JCIDS)  process.  .  .  .  The  AoA  is  one  of  the  key  inputs  to  the  process  of  defining 
system  capabilities  set  forth  in  the  capability  development  documents.3 

The  second  is  the  Capability  Development  Document  (CDD),  which  describes  the  user 
requirements  that  are  relevant  to  the  material  solution  addressed  by  the  source  selection.  The 
user  community  develops  these  requirements.4  During  the  technology  development  phase 
leading  up  to  Defense  Acquisition  Board  (DAB)  Milestone  B,  Air  Force  system  engineers  use 
the  CDD  to  inform  their  efforts  to  develop  the  System  Requirements  Document  (SRD)  for  the 
program  relevant  to  the  source  selection. 

The  SRD  is  the  third  requirements  document  of  interest.  It  derives  system-level  func¬ 
tional  and  performance  requirements  from  “items  such  as  the  CONOPS  [concept  of  opera¬ 
tions],  system-level  performance  metrics,  mission  threads/use  cases,  and  usage  environment” 
reflected  in  the  CDD.  The  SRD  reflects  a  “system  solution  defined  at  the  system  and  subsystem 
level  and  supported  by  [critical  technology  elements]  and  prototyping  results.  The  contractor 
uses  these  system  requirements  to  develop  the  system-specification  and  functional  and  perfor¬ 
mance  requirements  necessary  to  conduct  initial  end  item  design.”5 

The  SRD  is  usually  included  in  the  RFP  for  a  source  selection.  When  we  view  the  descrip¬ 
tion  above  in  the  context  of  a  source  selection,  the  “contractor”  becomes  an  offeror.  In  a  source 
selection,  the  SRD  defines  the  technical  performance  requirements  of  a  system  that  each  offeror 
must  address  in  its  proposal.  If  the  SRD  more  effectively  instantiates  the  intent  of  the  CDD 
drafted  by  the  user  community,  the  source  selection  using  the  SRD  is  probably  more  likely  to 
choose  the  proposal  that  best  satisfies  the  user’s  needs. 

Recent  policy  reviews  have  found  that  the  processes  that  develop  the  CDD  and  SRD  and 
those  that  choose  a  winner  within  a  source  selection  are  not  as  well  aligned  and  coordinated  as 
they  could  be.6  Our  interviews  with  Air  Force  personnel  revealed  broad  appreciation  of  these 
problems  and  a  strong  desire  within  the  acquisition  community  to  address  them.  In  the  context 
of  bid  protest  policy,  better  integration  of  the  requirements  determination,  technology  develop¬ 
ment,  and  source  selections  could  potentially  clarify  the  priorities  of  warfighters  in  ways  that 
allow  source  selections  to  target  discriminators  more  effectively  and  refine  more  clearly  how 
evaluation  would  treat  the  criteria  included  in  any  trade  space.7 


3  Marc  Greenberg  and  James  Gates,  Analysis  of  Alternatives,  Teaching  Note,  Ft.  Belvoir,  Va.:  Defense  Acquisition  Univer¬ 
sity.  For  information  on  the  JCIDS  process,  see  U.S.  Department  of  Defense,  2009f. 

4  The  CDD  is  one  product  of  JCIDS.  For  information  on  how  the  CDD  emerges  from  it,  see  U.S.  Department  of  Defense, 
“Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,”  Washington,  D.C.,  July  31, 
2009f. 

5  The  discussion  in  this  paragraph  is  based  on  Judith  S.  Dahmann  and  Mike  Kelley,  Systems  Engineering  During  the  Mate¬ 
riel  Solution  Analysis  and  Technology  Development  Phases,  white  paper,  Washington,  D.C.:  Office  of  the  Director,  Defense 
Research  and  Engineering,  System  Engineering,  September  2009.  Quotations  come  from  this  source. 

6  See,  for  example,  Christie,  Davis,  and  Porter,  2009;  Ronald  Kadish  (Lt  Gen,  USAF,  Ret.),  Defense  Acquisition  Perfor¬ 
mance  Assessment:  Report  by  the  Assessment  Panel  of  the  Defense  Acquisition  Performance  Assessment  Project  for  the  Deputy  Sec¬ 
retary  of  Defense,  Washington,  D.C.,  January  2006;  and  U.S.  Air  Force,  2009a. 

7  A  “trade  space”  means  different  things  at  different  points  during  a  program’s  life.  The  trade  space  addressed  during  the 
requirements  phase  is  broad.  It  includes  all  attributes  relevant  to  the  decision  on  what  requirements  to  set  for  a  new  system. 
The  trade  space  addressed  during  a  source  selection  includes  only  those  attributes  that  an  offeror  can  vary  to  add  value  to  its 
proposal.  At  one  extreme,  a  minimal  trade  space  in  a  source  selection  would  specify  threshold  values  for  all  attributes,  offer 
no  credit  for  any  performance  beyond  these  thresholds,  and  choose  among  proposals  that  achieve  these  requirements  on  the 
basis  of  other  grounds,  such  as  cost  or  past  performance.  An  extensive  trade  space  would  give  an  offeror  credit  for  increases 
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Development  and  Approval  of  a  Source  Selection  Plan 

The  source  selection  itself  begins  when  it  assembles  the  people  who  will  conduct  it  and  starts 
to  identify  how  to  frame  the  source  selection  as  a  whole.  The  primary  players  who  must  be 
identified  are 

•  The  SSA,  who  is  responsible  for  the  proper  and  efficient  conduct  of  the  source  selec¬ 
tion  process.  This  person  establishes  the  source  selection  team,  which  includes  the  SSET, 
SSAC,  and  additional  advisors.  He  or  she  monitors  the  source  selection,  approves  move¬ 
ment  past  significant  milestones  in  it,  and  ultimately  chooses  the  winner. 

•  The  contracting  officer,  who  manages  all  business  aspects  of  the  acquisition.  This  person 
has  day-to-day  responsibility  for  managing  the  contractual  details  of  the  source  selection. 
He  or  she  advises  SSA  on  every  event  in  source  selection. 

•  The  chair  of  the  SSET,  who  appoints  SSET  members  subject  to  SSA  approval.  This  person 
establishes  the  Performance  Confidence  Assessment  Group  (PCAG)  to  assess  the  past 
performance  of  offerors  in  all  source  selections  in  excess  of  $100  million. 

•  The  group  leaders  on  the  SSET  responsible  for  each  factor  that  must  be  evaluated.  The 
SSET  includes  technical,  service,  and  programmatic  evaluators;  contracting  officers  and 
buyers;  the  PCAG;  cost  or  price  analysts;  and  advisors. 

•  The  SSAC,  a  group  of  senior  advisors  used  on  most  large  source  selections  who  help  the 
SSA  monitor  the  source  selection  and  help  frame  information  moving  from  evaluation 
activities  to  the  SSA.  The  SSAC  includes  primary  stakeholders  in  acquisition,  including 
the  users.  The  SSAC  chair  appoints  SSAC  members  subject  to  SSA  approval. 

To  determine  how  to  frame  the  source  selection,  the  Air  Force  engages  in  a  variety  of 
activities,  including  the  following: 

•  Market  research  to  identify  the  technological  options  likely  to  become  available  to  the 
Air  Force  in  the  source  selection  and  the  attributes  of  these  options  that  are  most  likely  to 
distinguish  one  proposal  from  another  on  issues  that  interest  the  Air  Force. 

•  Risk  assessment  to  identify  which  attributes  deserve  particular  attention  as  discriminators 
and  what  forms  of  source  selection  can  help  the  Air  Force  mitigate  risks  identified.  This 
assessment  considers  potential  problems  that  can  impose  technological,  management, 
and  financial  risks  on  the  government. 

•  Cost  analysis  to  choose  costing  methods  and  begin  the  development  of  government  cost 
estimates  that  the  Air  Force  can  use  to  assess  the  realism  of  cost  estimates  in  proposals. 

•  Interaction  with  industry,  through  industry  days,  structured  communications,  and  ulti¬ 
mately  draft  requests  for  proposal,  to  gauge  their  capabilities  and  preferences  about  the 
form  of  the  source  selection,  ensure  that  offerors  understand  the  Air  Force  approach,  and 


in  the  levels  of  many  attributes.  When  this  occurs,  information  from  the  analysis  performed  during  the  requirements  phase 
can  give  the  government  guidance  on  how  to  weight  different  attributes  during  the  valuation  phase  of  the  source  selection. 

Given  the  importance  of  requirements  determination  to  source  selections  that  effectively  reflect  the  warfighter’s  priorities, 
one  Air  Force  reviewer  asked  us  why  we  had  not  given  more  attention  to  these  issues  in  this  report.  We  replied  that  we  had 
not  found  enough  information  about  it  to  support  significant  analysis.  He  responded  that,  until  someone  gives  these  issues 
more  attention,  there  will  never  be  enough  information  to  support  analysis.  We  agree  and  believe  that  this  area  deserves 
significantly  more  analytic  attention  in  the  future  than  it  has  received  to  date. 
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build  a  work  breakdown  structure  (WBS)  that  the  government  can  use  effectively  to 
compare  offers.8 

•  Development  and  approval  of  an  acquisition  plan  and  a  source  selection  plan  based  on 
what  the  Air  Force  learns  in  the  activities  above.  The  source  selection  plan  formalizes 
the  Air  Force  decisions  that  flow  from  those  activities.  The  plan  addresses  the  follow¬ 
ing  types  of  issues:  (1)  requirements,  (2)  acquisition  strategy,  (3)  source  selection  team, 

(4)  market  research,  draft  solicitations,  synopsis,  and  other  presolicitation  activities, 

(5)  process  of  communication  between  industry  and  government  and  within  government, 

(6)  evaluation  factors,  subfactors,  their  relative  importance,  and  the  evaluation  process,  and 

(7)  schedule. 

These  activities  closely  inform  the  environment  for  a  source  selection.  In  doing  so,  they 
affect  the  likelihood  of  a  protest  and  the  Air  Force’s  ability  to  justify  its  approach  to  source 
selection  if  a  protest  occurs.  In  addition,  the  Air  Force  preserves  draft  RFPs  and  the  source 
selection  plan  following  a  source  selection;  they  are  available  to  offerors  as  they  pursue  any 
protest.9 

Development,  Approval,  and  Announcement  of  an  RFP 

The  RFP  is  the  most  important  government  document  in  a  bid  protest.  Its  language  defines 
what  the  government  will  do  over  the  course  of  the  source  selection.  If  the  Air  Force  discov¬ 
ers  that  its  RFP  will  not  allow  it  to  choose  the  proposal  it  really  prefers,  it  must  either  amend 
the  RFP  or  settle  for  a  choice  that  does  not  give  it  what  it  wants.  Substantively  amending  an 
RFP  during  a  source  selection  is  fraught  with  difficulties.  Any  change  that  appears  to  favor  one 
offeror  over  others  can  easily  create  an  opening  for  other  offerors  to  protest. 

As  the  Air  Force  builds  an  RFP  that  complies  with  its  acquisition  and  source  selection 
plans,  it  gives  special  attention  to  Sections  L  and  M.  Section  L  defines  what  information  the 
Air  Force  wants  each  offeror  to  submit  in  its  proposal.  Section  M  defines  the  factors  relevant  to 
the  source  selection  and  how  the  Air  Force  will  evaluate  them.  In  particular,  Section  M  must 
identify,  for  example, 

•  whether  the  source  selection  will  use  a  trade-off  or  lowest-price  technically  acceptable 
evaluation  method 

•  how  the  source  selection  defines  its  mission  capability  technical  rating  and  technical  risk, 
past  performance,  cost  to  the  government  and  cost/price  risk,  and  any  other  factors  to  be 
used  in  the  evaluation 

•  the  relative  importance  of  all  factors  and  subfactors. 


8  A  WBS  breaks  a  program  into  pieces,  breaks  these  pieces  into  smaller  pieces,  and  so  on,  through  several  levels  of  inden¬ 
ture.  A  WBS  is  a  standard  tool  used  to  structure  the  management  of  a  program  or  to  estimate  program  costs.  Such  a  struc¬ 
ture  creates  a  set  of  accounts  that  can  help  the  government  ensure  that  it  captures  all  costs  in  a  comparable  way  and  does 
not  double  count  any  costs. 

9  U.S.  Air  Force,  2009b,  Sections  7.1-7.2;  U.S.  Department  of  Defense,  2009b,  Chart  71.  John  J.  Young,  Jr.,  Under  Sec¬ 
retary  of  Defense  (Acquisition,  Technology  and  Logistics),  established  the  SSJAT  in  November  2008  to  “standardize  the 
methodology  and  approach  in  conducting  source  selections”  to  ensure  that  the  DoD  systems  acquisition  process  “delivers 
the  highest  quality  products  to  our  Warfighters  and  the  best  value  to  the  taxpayers.”  Representatives  from  the  Directors 
of  Acquisition  Resources  and  Analysis  and  Portfolio  Systems  Acquisition,  the  Military  Departments,  the  OSD  General 
Counsel,  and  other  Defense  Agencies  staffed  three  integrated  policy  teams  responsible  for  “Understanding  the  Problem,” 
identifying  variations  in  practice  across  DoD,  and  identifying  “Best  Practices.”  U.S.  Department  of  Defense,  2009d,  p.  1. 
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Looking  at  source  selection  methods  used  across  DoD,  the  SSJAT  concluded  that,  when 
addressing  these  issues,  the  Air  Force  should  give  special  attention  to  the  following  questions: 10 

•  Are  the  contents  of  the  RFP  consistent  with  related  documents? 

•  Are  all  parts  of  the  RFP  internally  consistent  with  one  another? 

•  Given  the  approach  to  evaluation  defined  in  Section  M,  does  Section  L  ask  for  too  much 
or  too  little  information? 

•  Are  requirements  adequately  defined  in  the  SRD  and  elsewhere? 

•  Does  the  RFP  use  appropriate  contractual  incentives  to  deal  with  the  risks  identified 
earlier? 

Many  of  the  protests  sustained  in  the  CSAR-X  and  KC-X  programs  probably  would  not 
have  occurred  if  the  respective  source  selections  had  given  closer  attention  to  these  questions 
when  designing  their  RFPs.  The  Air  Force  preserves  the  RFP  following  a  source  selection;  it  is 
available  to  offerors  as  they  pursue  any  protest.* 11 

Evaluation  of  Proposals 

In  large  source  selections,  three  phases  of  evaluation  typically  occur: 

•  Exchanges  with  offerors  and  evaluation  to  establish  a  competitive  range.  This  determines 
whether  each  offeror  has  a  reasonable  chance  of  winning  the  competition. 

•  Exchanges  with  offerors  and  evaluation  within  the  competitive  range,  leading  to  final 
proposal  revisions  (FPRs).  This  gives  the  Air  Force  its  primary  opportunity  to  negotiate 
with  each  offeror  to  bring  its  proposal  as  close  as  possible  to  what  the  Air  Force  wants. 

•  Evaluation  associated  with  the  FPRs.  This  provides  the  evaluation  that  ultimately  decides 
how  closely  the  attributes  of  each  proposal  match  the  Air  Force  goals  stated  in  the  RFP. 

Differences  in  the  purposes  of  these  three  phases  lead  to  differences  in  how  the  Air  Force 
executes  them.  And  GAO’s  authority  to  assess  Air  Force  behavior  and  decisions  differs  some¬ 
what  in  each  round.  But  each  round  typically  involves  a  similar  series  of  events: 

•  give  offerors  information  on  their  evaluation  in  any  previous  round 

•  identify  strengths,  weaknesses,  deficiencies,  and  uncertainties  associated  with  each 
proposal 

•  communicate  with  each  offeror  about  these  and  negotiate  to  improve  each  proposal  where 
possible 

•  evaluate  the  proposals  as  they  stand  at  the  close  of  each  round. 

The  Air  Force  is  more  likely  to  avoid  sustained  protests  or  the  need  for  corrective  action 
in  each  round  if  it 

•  executes  evaluation  exactly  as  it  said  it  would  in  the  RFP 

•  treats  all  offerors  equally 


10  U.S.  Department  of  Defense,  2009b,  Chart  22. 

11  U.S.  Air  Force,  2009b,  Section  7.3;  U.S.  Department  of  Defense,  2009b,  Chart  71. 
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•  protects  information  in  each  offer  from  all  other  offerors 

•  makes  reasonable  judgments  and  decisions  through  the  course  of  each  round 

•  documents  these  judgments  and  decisions  clearly  in  real  time  through  the  course  of  each 

round. 

A  problem  associated  with  each  of  these  points  led  to  at  least  one  ground  for  sustaining  a 
protest  in  the  CSAR-X  and  KC-X  source  selections.  The  Air  Force  preserves  offeror  proposals, 
evaluation  worksheets,  interim  ratings,  competitive  range  determination  documents,  and  ENs 
following  a  source  selection;  they  are  available  to  offerors  as  they  pursue  any  protest.12 

Selection,  Approval,  and  Announcement  of  a  Winner 

This  final  stage  of  the  source  selection  involves  an  especially  elaborate  dance  among  the  players 
to  verify  that  the  choice  of  a  winner  is  justified  and  then  to  explain  the  basis  for  the  choice  as 
clearly  as  possible.  The  SSET  presents  the  results  of  its  FPR  evaluations  to  the  SSAC.  The  SSAC 
offers  one  consensus  source  selection  recommendation  for  the  SSA’s  consideration.  If  SSET 
members  seriously  disagree  on  evaluation  results,  the  SSET  presents  any  minority  opinions  to 
the  SSAC  and  SSA.  If  any  SSAC  members  disagree  on  what  recommendation  to  present  to 
the  SSA,  the  SSAC  presents  minority  views  to  the  SSA.  In  large  source  selections,  the  SSET 
chair  and  contracting  officer  prepare  a  Proposal  Analysis  Report  (PAR),  which  documents  the 
results  of  the  SSET  evaluation  and  comparison  of  offers,  including  its  assessment  of  cost  to  the 
government,  past  performance,  mission  capability,  cost/price  risk,  source  selection  recommen¬ 
dation,  and  any  minority  opinions. 

The  SSA  reviews  this  material  and  then,  using  independent  judgment  and  whatever  other 
information  may  be  available,  chooses  the  winner.  The  SSET  chair,  contracting  officer,  and 
other  members  of  the  source  selection  team  collaborate  to  document  the  basis  for  the  SSA’s 
decision  in  the  Source  Selection  Decision  Document  and,  when  the  decision  can  be  publicly 
announced,  the  SSA  uses  this  document  to  justify  his  or  her  decision.  This  is  a  single,  summary 
document  supporting  the  selection  of  the  best-value  proposal  consistent  with  the  stated  evalu¬ 
ation  criteria,  clearly  explaining  the  decision  and  documenting  the  reasoning  used  by  the  SSA. 
The  Air  Force  then  typically  debriefs  the  losers  to  explain  to  them  the  basis  of  the  decision. 

GAO  typically  gives  a  loser  broad  latitude  to  use  the  debriefing  to  conclude  that  an  error 
has  occurred.  GAO  gives  losers  ten  days  to  hie  any  protests  following  the  debriefing.  But  a  last 
opportunity  can  arise  if  information  previously  unknown  to  the  loser  appears  in  the  agency 
record  sent  to  the  source  selection  participants  at  the  close  of  the  source  selection.  A  loser  has 
ten  days  beyond  this  date  to  hie  a  protest. 

Following  the  decision,  the  Air  Force  preserves  the  following  documents  from  this  stage 
of  the  source  selection:13 

•  clearance  documentation 

•  decision  briehng 

•  PAR 

•  Source  Selection  Decision  Document 

•  source  selection  debriehng  documents 


12  U.S.  Air  Force,  2009b,  Sections  7.4— 7.7;  U.S.  Department  of  Defense,  2009b,  Chart  71. 

13  U.S.  Air  Force,  2009b,  Sections  7.8-7.13;  U.S.  Department  of  Defense,  2009b,  Chart  71. 
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•  artifacts  or  other  documentation  from  the  evaluation  process  (for  example,  individual 
evaluator  worksheets,  or  ENs  and  the  offerors’  responses). 

The  intricacies  of  this  last  stage  contributed  to  at  least  two  grounds  for  sustained  protests  in  the 
KC-X  source  selection.  All  of  the  documents  preserved  can  become  available  for  consideration 
in  a  protest.  This  illustrates  how  important  this  last  stage  is  to  any  effort  to  avoid  sustained 
protests. 
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How  Source  Selections  and  Bid  Protests  Work 

•Air  Force  source  selection 

•  GAO  bid  protest 

-  Initiation 

-  Initial  GAO  response 

-  Initial  Air  Force  response 

-  GAO  merit  review 

-  GAO  decision 
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Steps  of  the  Bid  Protest  Process 

GAO  runs  the  bid  protest  process  within  a  tight  schedule.  An  offeror  can  file  a  protest  at  any 
time  that  it  detects  an  error  in  the  source  selection  (typically  no  more  than  “10  days  after  the 
protester  knew  or  should  have  known  the  basis  of  protest”14).  From  the  date  of  that  filing, 
GAO  has  100  days  to  review  the  purported  error  and  to  render  a  judgment  on  it.15  GAO  rarely 
fails  to  meet  this  statutorily  mandated  schedule.16  This  is  true  in  part  because  it  imposes  heavy 
responsibilities  on  the  Air  Force  to  address  issues  raised  in  a  protest  and  respond  within  tight 
time  lines  of  its  own. 

The  bid  protest  process  usually  includes  the  following  steps:17 

•  initiation 

•  initial  GAO  response 

•  initial  Air  Force  response 

•  GAO  merit  review 

•  GAO  decision. 


14  U.S.  Government  Accountability  Office,  2006,  “When  to  Protest.”  An  important  exception  involves  errors  identified 
during  debriefings;  we  discuss  this  below. 

16  An  “express  option”  is  available  to  pursue  an  even  faster  schedule.  We  discuss  the  100-day  option  here. 

16  U.S.  Government  Accountability  Office,  2009. 

17  This  discussion  draws  heavily  on  Kayes,  2008;  U.S.  Government  Accountability  Office,  2006,  n.d.,  and  2009. 
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Initiation 

To  file  a  protest,  an  offeror  must  be  an  ‘“interested  party,’  which  means  an  actual  or  prospec¬ 
tive  bidder  or  offeror  with  a  direct  economic  interest  in  the  procurement.  ...  In  challenges  of 
the  evaluation  of  proposals  and  the  award  of  contracts,  this  generally  means  an  offeror  that 
would  potentially  be  in  line  for  award  if  the  protest  were  sustained.”18  “Although  most  protests 
challenge  the  acceptance  or  rejection  of  a  bid  or  proposal  and  the  award  or  proposed  award  of 
a  contract,  GAO  considers  protests  of  defective  solicitations  (e.g.,  allegedly  restrictive  specifica¬ 
tions,  omission  of  a  required  provision,  and  ambiguous  or  indefinite  evaluation  factors),  as  well 
as  certain  other  procurement  actions  (e.g.,  the  cancellation  of  a  solicitation).”19 

To  file  a  protest,  an  offeror  must  explain  why  GAO  has  jurisdiction  over  the  matter  in 
question,  explain  clearly  the  legal  basis  for  the  protest,  and  provide  enough  information  to  sup¬ 
port  a  review  of  the  claim.  To  help  broaden  access  to  its  review,  GAO  does  not  require  that  the 
offeror  hire  an  attorney  to  do  this  or  comply  with  precise  legal  forms.  But  the  GAO  process 
has  a  strong  legal  flavor  and  follows  legal  customs  of  truth  finding,  application  of  statutory 
authority  to  the  facts  in  any  specific  new  case,  and  building  the  position  for  a  decision  in  any 
such  case  on  precedent  set  by  previous  GAO  decisions.  So  skilled  attorneys  can  help  an  offeror 
a  great  deal  when  it  seeks  to  file  a  protest. 

Initial  GAO  Response 

When  an  offeror  files  a  protest  with  GAO,  GAO  notifies  the  Air  Force  within  a  day.  It  may 
then  assess  whether  the  offeror  has  a  right  to  file  a  protest.  Is  it  an  interested  party?  Does  GAO 
have  jurisdiction  over  the  issue  in  question?  Is  the  protest  timely,  relative  to  GAO  rules?  Does 
it  “set  forth  a  detailed  statement  of  the  legal  and  factual  grounds  of  protest”?  Does  it  “clearly 
state  legally  sufficient  grounds  of  protest”?20  GAO  can  “summarily  dismiss”  a  protest  on  any 
of  these  grounds. 

If  it  does  not  summarily  dismiss  the  protest,  GAO  notifies  the  Air  Force  that  it  has  30 
days  to 

provide  GAO,  the  protester,  and  any  intervenors  a  complete  written  report  responding  to 
the  protest,  including  all  relevant  documents,  or  portions  of  documents,  and  an  explana¬ 
tion  of  the  agency’s  position.  .  .  .  The  report  generally  includes  a  statement  of  the  relevant 
facts  (and  a  best  estimate  of  the  contract  value)  signed  by  the  contracting  officer,  a  memo¬ 
randum  of  law  explaining  the  agency’s  position  in  terms  of  procurement  law,  and  a  list  and 
a  copy  of  all  relevant  documents,  or  portions  of  documents,  not  previously  furnished.  .  .  . 

GAO  encourages  agencies  to  voluntarily  release  to  the  parties  documents  which  are  rel¬ 
evant  to  the  protest  prior  to  the  filing  of  the  agency  report.  .  .  .  GAO  requires  that  at  least 
5  days  prior  to  the  filing  of  the  report,  in  cases  in  which  the  protester  has  requested  in  its 
protest  or  shortly  thereafter  specific  documents  material  to  the  disposition  of  the  protest, 
the  agency  prepare  a  list  of  those  documents,  or  portions  of  documents,  which  it  has  previ¬ 
ously  released  or  intends  to  produce  in  its  report,  and  of  the  documents  which  it  intends  to 
withhold  and  the  reasons  for  the  proposed  withholding.21 


18  U.S.  Government  Accountability  Office,  2006,  “Who  May  Protest.” 

19  U.S.  Government  Accountability  Office,  2006,  “What  to  Protest.” 

20  U.S.  Government  Accountability  Office,  2006,  “Summary  Dismissal.” 

21  U.S.  Government  Accountability  Office,  2006,  “Agency  Report.” 
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initial  Air  Force  Response 

Upon  receiving  notice  of  a  protest  from  GAO,  the  Air  Force  immediately  notifies  any  com¬ 
pany  that  has  won  a  contract  under  the  competition  or  who  might  reasonably  expect  to  win  a 
contract  in  the  future.  GAO  may  permit  such  firms  to  participate  as  “intervenors.”  If  it  agrees 
that  an  error  has  occurred  and  could  have  large  enough  effects  to  change  its  choice  of  a  winner, 
the  Air  Force  can  offer  the  protester  corrective  action.  For  example,  the  Air  Force  may  cancel 
a  solicitation,  repeat  it,  reopen  it,  or  reevaluate  proposals  to  remove  errors.  The  next  chapter 
discusses  how  frequently  such  events  have  occurred.  If  the  corrective  action  offered  satisfies  the 
protester,  the  protest  ends  without  involving  GAO  further.  GAO  can  provide  flexible  alter¬ 
native  dispute  resolution  support  to  the  Air  Force  and  protester  to  encourage  moves  in  this 
direction.  If  the  Air  Force  and  protester  reach  an  agreement,  the  protester  either  withdraws 
its  protest  or  GAO  dismisses  it  as  “academic,”  because  no  further  GAO  action  can  affect  the 
outcome.  If  the  Air  Force  believes  that  the  protester  has  no  legitimate  grounds  to  protest,  it 
can  petition  GAO  for  summary  dismissal,  which  GAO  may  grant  based  on  the  Air  Force’s 
argument,  even  if  it  failed  to  do  so  earlier.  (An  intervenor  can  similarly  request  summary  dis¬ 
missal.)  If  the  Air  Force  thinks  no  error  has  occurred  that  is  large  enough  to  change  its  choice 
of  a  winner,  it  must  justify  this  judgment  and  send  GAO  and  the  protester  the  documents 
described  above. 

As  the  Air  Force  weighs  these  options,  it  can  ask  GAO  to  predict  how  it  might  rule  in  a 
merit  review  to  help  the  Air  Force  decide  what  to  do.  GAO  may  or  may  not  agree  to  do  this. 

GAO  gives  the  protester  ten  days  to  respond  to  any  agency  documents.  That  response 
can  come  in  the  form  of  an  additional  protest.  One  protest  can  lead  to  an  Air  Force  response, 
which  generates  documents  that  the  protester  can  use  to  file  another  protest.  In  the  same  way, 
a  set  of  protests  can  give  the  protester  access  to  Air  Force  documents  that  provide  the  basis  for 
another  set;  one  wave  begets  another.  This  process  can  generate  multiple  waves,  with  GAO 
starting  another  clock  on  each  new  set  of  protests. 

This  exchange  of  documents  can  generate  large  quantities  of  proprietary  and  otherwise 
sensitive  documents  traveling  among  all  the  participants  in  the  protest.  If  asked,  GAO  can 
issue  a  protective  order  that  defines  the  terms  under  which  these  parties  may  use  these  materi¬ 
als.  Typically,  the  immediate,  individual  participants  in  a  protest  may  not  share  these  materials 
with  their  principals.  For  example,  the  attorney  representing  a  protester  may  not  share  docu¬ 
ments  obtained  from  the  government  or  an  intervenor  with  others  in  the  protesting  company; 
the  protective  order  raises  a  Chinese  Wall  between  the  parties  privy  to  this  protected  informa¬ 
tion  and  everyone  else  beyond  the  wall.22  The  protective  order  defines  how  this  information 
will  be  protected  when  the  protest  ends.23 

GAO  Merit  Review 

When  it  becomes  apparent  that  a  protester  will  not  withdraw  and  has  a  legitimate  right  to  pro¬ 
test,  GAO  moves  toward  a  merit  review.  It  gathers  information  relevant  to  the  protests  in  play. 
It  allows  the  protester  to  discover  certain  classes  of  documents  in  government  hands;  it  is  much 
more  restrictive  in  its  willingness  to  allow  the  government  to  discover  the  protester. 


22  In  the  context  of  a  bid  protest,  “the  government”  typically  refers  to  the  government  agency  running  the  relevant  source 
selection.  That  is  how  we  use  “the  government”  in  this  discussion. 

23  Such  protective  orders  place  severe  constraints  on  access  to  such  material  by  third  parties  (i.e.,  researchers)  after  the  pro¬ 
test  ends. 
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If  the  protest  raises  complex  enough  issues  of  law  or  fact,  GAO  has  the  option  of  schedul¬ 
ing  hearings  to  gather  additional  information.  The  GAO  attorney  conducting  GAO’s  review  of 
the  protest  serves  as  the  hearing  official.  The  protester,  agency,  or  intervenors  can  also  request 
hearings.  Such  hearings  are  tailored  to  the  needs  of  each  case  and  range  widely  in  their  degree 
of  formality.  GAO  sets  the  agenda  for  any  hearing  and  can  decide  which  protest  issues  to  high¬ 
light  in  a  hearing.  No  hearing  is  required  for  GAO  to  rule  on  it,  for  or  against.  Protesters  have 
the  right  to  cross-examine  Air  Force  witnesses  who  participate  in  the  hearing.  The  Air  Force’s 
rights  are  much  more  constrained. 

GAO  Decision 

GAO  resolves  each  merit  protest  with  a  formal  written  decision  that  either  sustains  or  denies 
a  protest.  If  an  offeror  hies  20  grounds  for  protests,  and  GAO  sustains  only  one,  GAO  labels 
the  acquisition  where  this  protest  sustainment  occurred  to  have  a  “sustained  protest.”  The 
GAO  decision  documents  the  basis  for  its  sustainment  decision,  usually  framing  the  decision 
in  terms  of  earlier  GAO  decisions.  Taken  together,  the  body  of  sustained  protests  in  effect 
documents  the  factors  that  GAO  considers  when  reviewing  each  new  protest.  GAO  offers  a  list 
of  these  sustained  protests  as  a  document  that  interested  parties  can  use  to  assess  the  circum¬ 
stances  under  which  GAO  is  likely  to  sustain  a  protest  in  the  future.24 

GAO  formally  explains  the  basis  for  sustaining  every  ground  for  protest  that  it  sustains 
in  its  decision.  If  GAO  sustains  any  one  ground  for  protest  in  an  acquisition,  it  may  or  may 
not  discuss  other  grounds  for  protest  in  the  acquisition  when  it  documents  that  sustainment. 
It  explains  some  decisions  to  deny  specific  grounds  for  protest.  But  it  is  typically  impossible 
to  determine  by  simply  reading  a  formal  GAO  decision  how  many  grounds  for  protest  GAO 
reviewed  in  the  same  acquisition  and  how  many  of  these  it  might  have  dismissed  or  denied. 
GAO  gives  no  reasons  for  such  dismissals  and  denials. 

If  GAO  fails  to  sustain  a  single  ground  for  protest  in  an  acquisition,  it  documents  this  fact 
in  its  decision  and  explains  why  it  denied  at  least  one  ground  for  protest.  But  again,  as  above, 
simply  reading  a  formal  GAO  decision  does  not  reveal  how  many  other  grounds  for  protest 
GAO  might  have  denied  or  dismissed  in  that  acquisition  or  why.  We  do  not  even  know  with 
certainty  whether  the  ground  for  protest  that  GAO  denied  is  even  the  most  important  ground, 
however  that  might  be  defined.  GAO’s  published  decisions  serve  primarily  as  precedents  that 
GAO  can  use  in  future  protests  and  that  government  agencies  can  use  to  guide  their  future 
behavior. 

If  the  protester,  an  intervenor,  or  the  Air  Force  disagrees  with  the  GAO  decision,  it  can 
ask  GAO  to  reconsider  the  decision  and  explain  why  it  believes  GAO  should  do  so.  GAO  can 
do  so  or  not,  at  its  own  discretion. 

GAO  allows  any  protester  that  wins  a  sustained  protest  to  recover  at  least  some  of  its  costs 
of  protesting  from  the  agency  that  made  the  relevant  error.  Following  a  sustainment  decision, 
GAO  works  with  such  a  protester  to  determine  what  level  of  costs  GAO  will  allow  the  protester 
to  recover. 


24  U.S.  Government  Accountability  Office,  2006.  GAO  offers  no  more  systematic  explanation  of  how  it  makes  protest 
decisions.  From  GAO’s  perspective,  it  uses  no  rules,  formal  or  informal,  to  inform  the  substance  of  its  decisions.  It  simply 
applies  its  statutory  authority  to  the  specific  facts  on  any  one  case.  Speaking  about  its  decisions  outside  the  context  of  any 
specific  case  would  be  inconsistent  with  such  a  philosophy.  For  more  systematic  statements  about  how  GAO  makes  protest 
decisions,  see  Vernon  J.  Edwards,  Source  Selection  Answer  Book,  2nd  ed.,  Vienna,  Va.:  Management  Concepts,  2006;  the 
website  Where  in  Federal  Contracting  organizes  GAO  decisions  into  categories  that  make  it  easier  to  discern  patterns. 
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When  GAO  sustains  a  protest,  it  offers  a  recommended  corrective  action  that  the  Air 
Force  can  take  to  correct  the  error  identified.  GAO  lacks  authority  to  enforce  its  recommenda¬ 
tion,  but  government  agencies  almost  always  follow  GAO  guidance.  If  they  do  not,  GAO  is 
required  to  report  this  to  Congress. 

If  a  protester  fails  to  convince  GAO  to  sustain  a  protest,  it  can  take  its  protest  forward  as  a 
more  formal  legal  action  before  the  U.S.  Court  of  Federal  Claims,  which  tends  to  be  somewhat 
more  favorable  to  protesters  than  GAO  is.  But  filing  a  case  before  the  court  is  considerably 
more  costly  and  takes  much  longer.  Government  agencies  cannot  take  cases  in  which  GAO 
sustained  a  protest  to  this  court;  they  must  simply  take  corrective  action  or  face  the  political 
consequences.25 


25  A  more  complete  analysis  would  examine  Air  Force  experience  with  bid  protests  in  the  courts.  That  was  not  within  the 
scope  of  this  study.  For  information  on  this  court  and  its  role  in  bid  protests,  see  U.S.  Court  of  Federal  Claims,  “About  the 
Court,”  n.d.,  and  "Guidelines  for  Procurement  Protest  Cases,”  n.d. 
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Patterns  for  Bid  Protests  During  FY  1991-2008 


Road  Map 

•  How  source  selections  and  bid  protests  work 

•  Patterns  for  bid  protests  during  FY1 991 -2008 

•  Protests  sustained  in  CSAR-X,  KC-X  source 
selections 

•  Synthesis  of  findings 

•  Potential  changes  in  policy  or  practice 
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We  now  turn  to  quantitative  evidence  on  the  Air  Force’s  experience  with  bid  protests  over 
the  last  two  decades.  Data  from  the  Air  Force’s  PACTS  administrative  database  allow  us  to 
document  basic  patterns  in  the  characteristics  of  protests  that  occurred  and  the  outcomes  that 
flowed  from  these  protests.  Data  from  DD350  and  FPDS-NG  allow  us  to  put  this  information 
on  protests  in  a  broader  context  and  ask  what  factors  beyond  the  source  selection  process  might 
have  affected  the  Air  Force’s  experience  with  bid  protests.  Statistical  analysis  of  data  from  these 
sources  allows  us  to  ask  how  individual  factors  have  affected  the  protests  that  the  Air  Force  has 
experienced  and  the  outcomes  they  led  to.  This  chapter  addresses  these  topics  in  turn.1 


1  Light  et  al.,  2011,  describes  the  databases  that  underlie  the  analysis  reported  here.  It  also  describes  in  much  greater  detail 
the  statistical  analysis  we  applied  to  material  from  these  databases  to  develop  the  findings  reported  here. 
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Protests  Result  from  Errors  Perceived 
Throughout  the  Source  Selection  Process 


Step  in  source  selection 

Reasons  for  protests 

Percent 

Develop  requirements 

Develop  sourcing  strategy 

Faulty  sourcing  action 

18% 

Develop  and  issue  RFP 

Faulty  RFP 

18% 

Conduct  discussions  with  offerors 

Faulty  treatment  of  offeror 

8% 

Evaluate  proposals 

Faulty  evaluation 

60% 

Choose  winner,  debrief  losers 

Other 

6% 

Note:  Frequency  counts  based  on  root  B-numbers  and  calculated  from  PACTS  data  for 
FY  2000-08.  Some  protests  with  multiple  B-numbers  are  associated  with  multiple  protest  reasons, 
causing  the  sum  of  protests  by  protest  basis  to  equal  more  than  the  total  number  of  protests. 
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General  Patterns  of  Air  Force  Experience  with  Bid  Protests 

This  slide  allows  us  to  examine  where,  in  the  source  selection  process  described  in  Chapter 
Two,  protesters  claimed  that  the  Air  Force  made  errors  during  the  period  FY  2000-2008. 2  The 
first  column  lists  steps  in  the  process  described  in  Chapter  Two.  The  second  column  matches 
the  reasons  for  protests  listed  in  the  PACTS  database  to  these  steps.  The  “other”  category  in 
this  column  includes  reasons  that  we  could  not  reliably  match  to  any  particular  step  in  the  first 
column. 

These  data  suggest  that  protesters  have  believed  that  errors  occurred  throughout  the 
source  selection  process.3  None  occurred  in  requirements  determination  because,  as  explained 
in  Chapter  Two,  this  is  beyond  GAO’s  acquisition-oriented  jurisdiction.  But  a  significant 
number  occurred  before  the  Air  Force  even  issued  an  RFP.  These  tended  to  involve  errors  in 
how  the  Air  Force  organized  a  competition  or  classified  a  protester  in  a  competition.  Protesters 
have  believed  that  the  lion’s  share  of  errors  occur  during  evaluation.  Errors  here  occurred  when 
the  Air  Force  failed  to  evaluate  a  proposal  the  way  the  RFP  said  it  would,  used  an  unreasonable 


2  For  definitions  of  these  categories,  see  discussion  at  Light  et  ah,  2011,  Table  2.1. 

3  We  need  to  approach  data  of  this  kind  with  some  caution.  As  with  the  data  found  in  many  administrative  databases, 
those  coming  from  fields  in  PACTS  that  are  not  used  frequently  to  inform  decisions  tend  to  be  fairly  “dirty” — that  is,  the 
entries  are  often  likely  not  to  be  correct  or  are  impossible  to  process  in  a  consistent,  systematic  way.  “Reason  for  protest” 
appears  to  be  one  such  field.  Patterns  of  reporting  appear  to  shift  over  time  as  the  personnel  recording  the  data  change, 
suggesting  some  ambiguity  in  what  labels  someone  working  today  might  have  recorded  if  he  or  she  had  recorded  the  data 
15  years  ago.  That  said,  we  believe  that  these  data  present  a  rough  idea  of  how  reasons  for  protests  occur  across  steps  of  the 
source  selection  process. 
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argument  to  justify  an  evaluation,  or  failed  to  record  its  argument  in  real  time  during  evalu¬ 
ation.  Content  analysis  of  recent  sustainments  in  Air  Force  acquisitions  indicates  that  these 
problems  have  dominated  errors  that  GAO  identified  in  its  protest  sustainments.4 


4  For  evidence  from  FY  2006-2008,  see  Kayes,  2008,  Charts  9-12. 
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Air  Force  Changed  a  Third  of  836  Source 
Selections  Protested  During  FY  2000-08 


Note:  Based  on  root  B-number  counts  for  FY  2000  -  2008.  All  calculations  derived  from  PACTS 


RAND 
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This  slide  allows  us  to  analyze  the  GAO  bid  protest  process  in  action  during  the  period 
FY  2000-2008. 5  Over  that  period,  the  Air  Force  experienced  protests  in  836  acquisitions  with 
contract  awards.  It  allows  us  to  sort  protests  by  four  policy-relevant  types  of  outcomes.  First, 
in  a  third  of  the  protests — 273 — the  Air  Force  negotiated  voluntary  corrective  actions.  All  of 
these  protests  were  withdrawn  by  the  protester  or  dismissed  by  GAO.  Second,  362  additional 
protests — 43  percent — were  withdrawn  by  the  protester  without  corrective  action  or  dismissed 
by  GAO  because  they  were  inappropriate  for  some  reason  (we  examine  reasons  for  dismissals 
below).  All  other  protests  went  to  merit  review.  Third,  of  those  that  experienced  merit  review, 
GAO  denied  172  protests — 86  percent  of  those  subject  to  merit  review  and  21  percent  of  the 
total.  Fourth,  GAO  sustained  29 — 14  percent  of  those  experiencing  merit  review  and  only  3 
percent  of  the  total. 

All  of  these  types  of  protests  impose  some  costs  on  the  Air  Force.  Some  administrative 
and  legal  burden  accompanies  each  of  them.  This  burden  rises  when  a  protest  enters  a  merit 
review.  Because  merit  reviews  consume  calendar  time,  they  also  impose  delays  that  can  com¬ 
plicate  Air  Force  resource  and  operational  planning.  The  most  significant  costs  accompany 


’  These  numbers  are  constructed  from  data  reported  in  PACTS.  A  protest  is  considered  a  merit  protest  if  at  least  one 
associated  B-number  is  sustained  or  denied.  A  merit  protest  is  considered  sustained  if  at  least  one  associated  B-number  is 
sustained;  otherwise  it  is  denied.  A  protest  is  considered  dismissed  if  at  least  one  non-merit  protest  B-number  is  dismissed; 
otherwise  it  is  considered  withdrawn.  Eleven  protests  in  PACTS  reported  as  closed  are  considered  dismissed  for  the  pur¬ 
poses  of  these  tabulations.  Definitions  for  actions  and  decisions  shown  in  the  slide  appear  to  be  stable  over  time  and  well 
understood.  PACTS  appears  to  provide  fairly  accurate  data  on  these  categories.  Further  discussion  is  available  in  Light  et 
ah,  2011,  Figure  2.1. 
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corrective  actions  and  sustained  protests.  As  we  will  see  below,  these  can  require  significant 
changes  in  source  selection  that  induce  substantial  delays  and  associated  costs. 
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Corrective  Actions  During  FY2000-08 
Imposed  Substantial  Costs  on  USAF 


Corrective  Action 

Number 

Percent 

Cancel  solicitation 

50 

13% 

Repeat  solicitation 

95 

25% 

Reopen  discussions 

24 

6% 

Reevaluate 

146 

40% 

Other 

58 

15% 

Unknown 

5 

1% 

Total 

378 

100% 

Note:  Tabulated  by  B-number  (FY  2000-08)  from  PACTS  data. 
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Like  the  perceived  errors  associated  with  the  reasons  for  protest  shown  in  Slide  9,  the 
errors  that  result  in  voluntary  corrective  actions  appear  to  occur  at  all  stages  in  source  selec¬ 
tion.6  The  types  of  corrective  actions  listed  here  are  the  same  as  those  that  GAO  recommends 
when  it  sustains  a  protest.  The  costs  to  the  Air  Force  of  these  corrective  actions  are  likely  to  vary 
significantly.  The  dominant  form  of  corrective  action  shown  in  the  slide,  reevaluation,  imposes 
less  cost  on  the  Air  Force  than  the  others  detailed  here.  Reopening  discussions  allows  offerors 
to  adjust  their  proposals,  potentially  in  ways  that  can  significantly  complicate  Air  Force  evalua¬ 
tion  of  the  changes.  Repeating  a  solicitation  adds  still  more  administrative  cost  and  delay.  And 
cancellation  of  a  solicitation  can  have  much  longer  and  deeper  substantive  effects  on  Air  Force 
capabilities.  Recall,  in  the  case  of  the  CSAR-X  program,  that  protests  ultimately  helped  lead  to 
the  cancellation  of  the  program  itself. 


6  Note  that  this  slide  reports  numbers  based  on  B-numbers.  Because  the  last  slide  reported  numbers  based  on  root 
B-numbers,  the  total  associated  with  number  of  corrective  actions  differs  in  the  two  slides.  “Unknown”  applies  to  cases  in 
which  PACTS  says  that  a  corrective  action  occurred  but  does  not  specify  its  form.  We  are  cautious  about  our  confidence  in 
the  accuracy  of  PACTS  data  on  corrective  actions.  However,  viewed  at  this  level  of  aggregation,  we  believe  they  offer  useful 
insights  into  relative  numbers  of  different  kinds  of  corrective  actions.  No  historical  data  exist  on  the  actual  types  of  errors 
that  lead  to  such  corrective  actions.  Air  Force  Legal  Operations  Agency  (AFLOA)/Commercial  Litigation  Directorate  of  the 
Air  Force  Legal  Operations  Agency  (JAQ)  began  to  collect  and  preserve  such  data  in  early  2010. 
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GAO  Required  and  Voluntary  Corrective 
Actions  Impose  Costs  on  Air  Force 

•  No  solid  empirical  evidence  is  available  on  relative 
costs. 

*  Interviews  with  Air  Force  officials  suggest  that 

-  “Protest  costs”  are  higher  when  merit  reviews  occur  than 
when  they  do  not. 

-  “Corrective  action  costs”  are  about  the  same  whether  a 
merit  review  occurs  or  not. 

*  Economic  reasoning  suggests  that  costs  are  higher 
for  typical  GAO-required  than  for  typical  voluntary 
corrective  actions. 

•  But,  because  voluntary  corrective  actions  have 
been  so  much  more  common,  they  are  important 
too. 

MD 
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In  any  particular  protest,  how  do  the  costs  to  the  Air  Force  compare  for  (1)  offering  a  vol¬ 
untary  corrective  action  and  (2)  going  forward  to  merit  review  and  potentially  a  GAO-induced 
corrective  action?  Consider  two  different  kinds  of  costs.  The  first  “protest  cost”  is  the  cost  of 
participating  in  the  protest  and  includes  administrative  costs  and  the  costs  of  distraction.  The 
second  “corrective  action  cost”  is  the  cost  to  the  Air  Force  of  executing  the  corrective  action 
and  includes  delay,  distraction,  and  dollar  costs. 

Air  Force  attorneys  told  us  that  the  costs  of  going  forward  to  merit  review  significantly 
exceed  the  costs  of  a  voluntary  corrective  action.  Delays  are  longer.  Although  a  proactive  cor¬ 
rective  action  can  often  be  taken  within  30  days  of  a  protest  filing,  a  written  decision  from 
GAO  requires  close  to  100  days.  And  to  some  extent,  the  Air  Force  can  negotiate  the  scope  of  a 
corrective  action  with  a  protester,  potentially  reducing  its  cost  to  the  Air  Force.  It  is  reasonable 
to  expect  at  a  minimum  that  protest  costs  will  be  significantly  lower  if  the  Air  Force  voluntarily 
offers  and  the  protester  accepts  corrective  action.  But  no  one  maintains  reliable  records  on  how 
much  lower  costs  have  actually  been. 

Most  Air  Force  acquisition  and  program  management  specialists  we  spoke  with  saw  less 
of  a  difference.  They  believe  that  corrective  action  costs  dominate  protest  costs  and  that  a  cor¬ 
rective  action  is  a  corrective  action;  corrective  action  costs  are  about  the  same  whether  the  Air 
Force  volunteers  a  corrective  action  or  GAO  requires  it.  This  would  suggest  that  corrective 
action  costs  are  about  the  same  whether  GAO  frames  the  corrective  action  or  the  Air  Force 
and  protester  do  so  before  a  merit  review.  But  again,  no  one  maintains  reliable  records  of  how 
high  either  is  in  practice. 

In  the  absence  of  any  solid  empirical  evidence  on  costs,  are  these  qualitative  observations 
consistent  with  what  we  can  observe  in  empirical  patterns  of  protests?  So  many  assumptions 
are  required  to  allow  such  a  comparison  that  we  must  be  cautious  about  basing  policy  judg- 
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ments  on  such  a  comparison.  But  roughly  speaking,  it  appears  reasonable  to  expect  that  correc¬ 
tive  action  costs  tend  to  be  higher  when  required  by  GAO  than  when  negotiated  voluntarily. 
The  reason  is  that,  as  the  expected  cost  of  a  corrective  action  grows  in  a  particular  protest,  it 
becomes  more  attractive  to  the  government  to  spend  the  resources  required  for  merit  review  to 
reduce  the  probability  that  it  must  undertake  corrective  action.  All  else  equal,  this  observation 
tells  us  that  the  Air  Force  will  tend  to  offer  voluntary  corrective  action  on  protests  with  lower 
costs  of  corrective  action  and  send  other  protests  directly  to  merit  review.7  Without  empiri¬ 
cal  information,  we  cannot  say  anything  more  about  how  much  higher  the  costs  of  corrective 
action  might  be  in  sustained  protests. 

Even  if  the  costs  for  corrective  action  in  individual  protests  are  significantly  higher  if  it 
results  from  a  GAO  decision  rather  than  from  voluntary  negotiation  between  the  Air  Force  and 
the  protester,  it  is  worth  keeping  in  mind  that,  during  FY  2000-2008,  voluntary  agreements 
were  over  nine  times  as  common  as  GAO-sustained  protests.  As  a  result,  even  if  the  average 
sustained  protest  imposes  larger  costs  on  the  Air  Force  than  the  average  voluntary  corrective 
action,  voluntary  corrective  actions  will  continue  to  impose  large  costs  on  the  Air  Force.  No 
matter  how  much  higher  the  costs  of  GAO-imposed  corrective  actions  are,  it  is  worth  tracking 
numbers  of  voluntary  corrective  actions  as  well  as  numbers  of  sustained  protests. 


^  Appendix  A  explains  this  argument  in  more  detail. 
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Protests  Dismissed  Without  Corrective  Action 
Suggest  the  Presence  of  Naive  Protesters 


GAO  Dismissal  Basis 

#  of  Dismissed 
Protest  B- 
Numbers 

%of 

Dismissed 
Protest  B- 
Numbers 

No  longer  relevant  (“academic”) 

299 

45% 

Lack  of  GAO  jurisdiction 

59 

9% 

Lack  of  standing 

35 

5% 

Lack  of  legal  basis 

66 

10% 

Failure  to  use  GAO  process 
properly 

160 

24% 

Other 

47 

7% 

Total 

666 

100% 

Note:  Derived  from  dismissal  reasons  codes  contained 
in  PACTS  and  tabulated  by  B-number  for  FY2000-08. 
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GAO  dismissed  over  60  percent  of  protests  of  Air  Force  source  selections  during  the 
period  FY  2000-2008. 8  The  basis  for  those  dismissals  points  to  the  persistent  presence  of  pro¬ 
testers  who  did  not  understand  the  GAO  protest  process  when  they  hied  a  protest.  In  about 
half  the  protests  that  occurred,  GAO  lacked  jurisdiction  over  the  issue  in  question,  the  pro¬ 
tester  lacked  standing  before  GAO,  the  protester  could  not  state  a  legal  basis  for  the  protest 
that  met  GAO’s  standards,  or  the  protester  used  GAO’s  protest  process  incorrectly.9  GAO  dis¬ 
missed  almost  all  the  rest,  because  it  judged  the  protests  to  be  academic — no  longer  of  interest 
because  they  no  longer  required  GAO’s  attention.  The  vast  majority  of  such  protests  became 
academic  because  the  Air  Force  and  protester  had  agreed  to  some  form  of  corrective  action  that 
resolved  the  protester’s  problem. 

These  results  point  to  two  observations.  First,  a  large  share  of  protests — about  a  third — 
is  in  effect  the  result  of  some  error  on  the  part  of  a  protester.  These  protests  are  beyond  the 
Air  Force’s  direct  control.  Unless  the  Air  Force — or  someone — can  educate  the  offerors  who 
participate  in  Air  Force  source  selections  on  the  standards  that  GAO  applies,  the  Air  Force 
can  expect  to  face  those  protests  for  the  indefinite  future.  These  are  the  protesters  that  GAO’s 
bid  protest  rules,  seeking  simplicity  and  low  administrative  costs  for  all  involved,  seek  most  to 
protect. 


8  Again,  the  total  number  of  dismissals  reported  here  (666)  differs  from  that  reported  in  Slide  10  (517)  because  Slide  10 
counts  root  B-numbers;  Slide  12  counts  B-numbers. 

9  These  results  are  constructed  from  data  in  PACTS.  We  are  cautious  about  our  confidence  in  the  accuracy  of  PACTS  data 
on  dismissals.  Viewed  at  this  level  of  aggregation,  we  believe  that  they  offer  useful  insights  into  relative  numbers  of  different 
kinds  of  corrective  actions. 
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Second,  although  such  protests  do  impose  administrative  and  legal  burdens  on  the  Air 
Force,  these  burdens  are  mainly  an  irritant  when  viewed  in  comparison  to  the  costs  of  sus¬ 
tained  protests  and  corrective  actions.  They  provide  unavoidable  workload  for  a  small  cadre  of 
Air  Force  personnel  that  unfortunately  adds  little  value  to  the  Air  Force  mission.  Sustainments 
and  corrective  actions  can  have  much  more  direct  effects  on  the  Air  Force  mission;  the  Air 
Force  can  help  the  warfighter  by  focusing  its  efforts  to  improve  its  source  selection  policies  and 
practices  on  these  outcomes  of  the  GAO  protest  process. 
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Bid  Protests  Are  Having  Steadily  Smaller 
Effects  on  Air  Force  Acquisition 


Note:  Based  on  root  B-numbers  and  calculated 
from  PACTS  and  FPDS-NG  data  for  FY  1994-2008. 


RAND 
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To  understand  protests  and  the  outcomes  they  can  lead  to,  it  helps  to  view  them  in 
the  context  of  the  total  number  of  contract  awards  that  the  Air  Force  makes.  This  slide  does 
that.  In  effect,  it  normalizes  the  number  of  protests  and  potential  protest  outcomes  that  have 
occurred  each  year  by  stating  them  relative  to  thousands  of  contract  awards.10  This  normaliza¬ 
tion  allows  us  to  make  the  following  observations. 

First,  shown  in  blue,  the  total  number  of  Air  Force  protests  as  a  share  of  total  number  of 
awards  fell  dramatically  from  FY  1994  to  FY  2008 — about  65  percent  or  about  7  percent  per 
year.  Why?  Suppose  we  define  a  protest  that  results  in  a  corrective  action  or  a  merit  review  as 
a  “substantive”  protest.  The  red  curve  shows  the  trend  for  the  number  of  substantive  protests 
as  a  share  of  total  contract  awards.  This  share  also  fell  fairly  steadily  from  FY  1994  to  FY  2008 
but  not  nearly  as  fast  as  total  protests.  The  difference  between  these  trends  is  the  number  of 
protests  in  which  the  protester  probably  made  an  error  filing.  The  number  of  such  nonsubstan¬ 
tive  or  inappropriate  protests  as  a  share  of  total  awards  fell  dramatically  until  about  FY  2000 
and  then  stabilized.  Since  FY  2001,  improvements  in  the  total  number  of  protests  have  come 
mainly  from  improvements  in  the  number  of  substantive  protests. 


10  This  slide  is  constructed  from  data  in  PACTS,  DD350,  and  FPDS-NG.  All  of  these  are  fairly  “dirty”  administrative 
databases.  Many  data  are  missing.  Those  that  are  present  are  entered  by  different  people  with  different  interpretations  of 
what  should  be  entered.  Many  display  simple  typographical  errors  that  confound  automated  data  analysis.  We  have  used 
versions  of  DD350  and  FPDS-NG  that  RAND  has  significantly  cleaned  and  maintained  over  time.  Viewed  at  this  level  of 
aggregation,  we  believe  the  qualitative  patterns  traced  using  these  data  are  valid.  We  could  have  used  the  number  of  com¬ 
petitive  contract  awards  to  normalize  these  outcomes.  We  explain  below  that,  because  the  numbers  of  total  contract  awards 
and  competitive  contract  awards  have  moved  together  so  closely  for  the  Air  Force,  the  choice  has  no  effect  on  the  qualitative 
character  or  policy  implications  of  the  patterns  that  we  report  here. 


34  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and  Options 


What  has  driven  down  the  number  of  substantive  protests?  The  number  of  corrective 
actions  per  contract  award,  shown  in  purple,  bobbed  around  until  FY  2000  and  then  began  a 
long  downward  trend  that  helps  explain  part  of  the  pattern  of  improvement  since  FY  2000.  The 
remaining  component  of  substantive  protests,  GAO  merit  reviews,  accounts  for  the  remainder 
of  improvement.  We  see  this  in  the  steady  drop  in  the  number  of  merit  protests  per  contract 
award,  shown  in  yellow.  The  number  of  sustainments  per  contract  award,  shown  in  green,  is 
so  small  that  it  displays  no  discernible  trend  over  this  period  of  time.  In  effect,  this  number  is 
constant. 

The  relationship  between  a  constant  number  of  sustained  protests  per  1,000  contract 
awards  and  a  falling  number  of  denied  protests  per  1,000  contract  awards  leads  to  an  observa¬ 
tion  that  the  share  of  merit  protests  that  the  Air  Force  loses  has  increased  over  time,  even  as 
the  number  of  merit  protests  per  1,000  contract  awards  has  fallen.  Too  much  focus  on  this 
observation  obscures  three  more  important  points:  (1)  The  total  number  of  sustained  protests 
per  1,000  contract  awards  has  not  risen.  (2)  As  noted  above,  the  number  of  corrective  actions 
has  heavily  dominated  the  number  of  sustained  protests,  and  their  number  per  1,000  contract 
awards  has  steadily  fallen.  (3)  The  number  of  merit  protests  per  1,000  contract  awards  that  the 
Air  Force  has  to  respond  to  has  been  falling. 

Taken  together,  these  observations  tell  a  positive  story  about  the  Air  Force’s  broad  experi¬ 
ence  with  GAO  bid  protests  in  recent  years. 
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Protests  Induce  Air  Force  to  Adjust  Only  a 
Small  Fraction  of  All  Source  Selections 


Portion  of  AF 

Metric 

Total  AF 

Percent  of  USAF  procurement  spend 

100 

Average  number  of  awards  per  year 

14,813 

Average  number  of  protests  per  year 

92.9 

%  inducing  corrective  actions 

33 

%  resulting  in  merit  protests 

24 

%  sustained 

3 

%  of  protests  requiring  AF  change 

36 

%  of  awards  requiring  AF  change 

0.23 

Note:  Protest  counts  based  on  root  B-number  counts  for  FY  2000-08.  Contract  expenditures  are 
reported  in  2008  dollars  using  the  GDP  price  deflator.  Data  from  PACTS,  DD-350,  FPDS-NG. 

RAND 
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By  focusing  on  numbers  of  sustained  protests  and  corrective  actions,  we  can  put  the  Air 
Force’s  broad  experience  with  bid  protests  in  perspective.  This  slide  explains  a  simple  method 
that  we  will  use  on  the  next  slide  to  look  for  differences  in  experience  with  bid  protests  across 
the  Air  Force  during  the  period  FY  2000-2008. 11  The  numbers  in  the  column  are  annual  aver¬ 
ages  over  this  period. 

The  slide  starts  with  the  proportion  of  the  average  total  Air  Force  spend  over  this  period. 
The  relevance  of  this  proportion  will  become  apparent  on  the  next  slide.  It  then  lists  the  aver¬ 
age  number  of  contracts  per  year  and  the  average  number  of  protests  per  year.  It  shows  that  33 
percent  of  the  92.9  protests  occurring  in  an  average  year  induce  corrective  actions,  24  percent 
result  in  merit  reviews,  and  3  percent  result  in  sustained  protests.  The  share  of  protests  resulting 
in  either  a  corrective  action  or  a  sustained  protest  is  36  percent.12  This  share  implies  that  the 
Air  Force  ultimately  had  to  change  0.23  percent  of  the  source  selections  for  all  of  its  contract 
awards  during  FY  2000-2008  in  response  to  GAO  bid  protests.  This  small  number  contrib¬ 
utes  to  the  positive  story  told  on  the  last  slide. 


11  The  numbers  reported  here  derive  from  the  same  data  sources  used  in  the  last  few  slides.  The  same  caveats  about  these 
data  sources  apply  here. 

12  Because  a  protest  with  a  corrective  action  listed  later  occasionally  leads  to  a  sustained  protest,  these  numbers  are  not 
strictly  additive.  Our  analysis  reflects  this  kind  of  overlap  in  outcomes.  Because  such  an  outcome  is  so  unusual,  though,  it 
does  not  affect  the  results  reported  at  the  level  of  precision  used  here. 
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Simple  Comparison  Reveals  Only  Modest 
Differences  over  Types  of  Contract  Centers 


Portion  of  AF 

Metric 

Total 

AF 

HQ 

Product 

centers 

ALC 

Other 

technical 

Projec¬ 

tion 

base 

Other 

base 

Percent  of  AF  spend 

100 

7 

52 

21 

5 

7 

7 

#  awards 
(thousands) 

14.8 

1.8 

1.5 

3.7 

1.7 

4.4 

1.8 

#  protests 

92.9 

11.4 

8.2 

15.4 

3.7 

33.2 

20.9 

%  corrective  actions 

33 

34 

34 

34 

22 

34 

30 

%  merit  protests 

24 

33 

29 

27 

19 

21 

22 

%  sustained 

3 

2 

10 

6 

0 

1 

5 

%  protests  requiring 
AF  change 

36 

36 

44 

40 

22 

35 

35 

%  source  selections 
requiring  AF  change 

0.23 

0.23 

0.25 

0.17 

0.05 

0.26 

0.41 

Note:  Protest  counts  based  on  root  B-number  counts  for  FY  2000-08.  Contract  expenditures  are 
reported  in  2008  dollars  using  the  GDP  price  deflator.  Data  from  PACTS,  DD-350,  FPDS-NG. 
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Does  the  Air  Force  experience  with  bid  protests,  viewed  in  this  way,  vary  in  different  parts 
of  the  Air  Force?  We  expect  differences  because  different  parts  of  the  Air  Force  buy  different 
kinds  of  goods  and  services,  use  different  kinds  of  contracts  and  source  selection  methods,  and 
maintain  different  contracting  cultures.  Knowledgeable  observers  generally  agree  that  the  level 
of  training  and  experience  of  the  personnel  involved  in  source  selections  also  varies  systemati¬ 
cally  across  the  Air  Force.  To  look  for  the  effects  of  such  differences  on  bid  protest  outcomes, 
we  grouped  the  contracting  activities  in  the  Air  Force  into  the  following  six  categories: 

The  headquarters  for  each  Air  Force  major  command  (MAJCOM)  and  Unified  Com¬ 
mand.  The  contracting  squadrons  of  headquarters  units  acquire  certain  kinds  of  goods  and 
services  that  are  used  by  all  bases  within  its  organization.  These  contracts  tend  to  be  larger 
and  more  complex  and  to  require  more  formal  source  selection  activity  than  those  at  indi¬ 
vidual  base  locations.  Examples  of  headquarters  units  included  here  are  Langley  Air  Force 
Base  (AFB),  Air  Combat  Command  (ACC);  Peterson  AFB,  Air  Force  Space  Command;  Scott 
AFB,  Air  Mobility  Command  (AMC);  Ramstein  Air  Base,  U.S.  Air  Force,  Europe;  Hickam 
AFB,  Pacific  Air  Forces;  Randolph  AFB,  Air  Education  and  Training  Command  (AETC);  and 
Cheyenne  Mountain,  U.S.  Strategic  Command. 

The  product  centers,  which  include  the  Aeronautical  Systems  Center,  Wright  Patterson 
AFB;  the  Electronic  Systems  Center,  Hanscom  AFB;  Air  Armament  Center,  Eglin  AFB;  and 
Space  and  Missile  Center,  Los  Angeles  AFB.  These  tend  to  buy  major  systems  and  the  activities 
associated  with  them,  using  relatively  highly  trained  and  experienced  work  forces. 

The  air  logistics  centers  (ALCs),  which  include  Oklahoma  ALC,  Tinker  AFB;  Ogden  ALC, 
Hill  AFB;  Warner  Robins  ALC,  Robins  AFB;  and,  before  2001,  Sacramento  ALC  (SM-ALC), 
McClellan  AFB;  San  Antonio  ALC  (SA-ALC),  Kelly  AFB;  and  the  Aerospace  Guidance  and 
Metrology  Center  (AGMC),  Newark  AFB.  As  a  result  of  several  base  realignment  and  closure 
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decisions  SM-ALC,  SA-ALC,  and  AGMC  and  their  associated  bases  were  closed.  ALCs  buy 
parts,  commodities,  and  logistics  services. 

Technical  contracting  centers  that  work  at  units  and  locations  conducting  test  and  evalu¬ 
ation  activities  at  Edwards,  Eglin,  or  Kirdand  AFBs  or  are  research  and  development  labs, 
such  as  the  Air  Force  Research  Lab,  which  includes  Philips  Lab,  Kirdand  AFB;  Wright  Lab, 
Wright  Patterson  AFB;  Rome  Lab,  Griffiss  AFB;  and  Armstrong  Lab,  Brooks  City  Base.  These 
use  highly  trained  and  experienced  personnel  to  buy  test  and  evaluation  services  and  related 
research  and  development  services. 

Projection  bases,  which  are  defined  as  individual  bases  or  locations  that  operate  weapon 
systems  having  a  wartime  mission  or  that  can  be  deployed.13  Except  for  AFMC,  bases  for  each 
MAJCOM  are  included.  Contracts  written  by  projection  bases  provide  base  installation  or 
base  operating  support.  They  often  have  smaller  dollar  values  and  require  fewer  formal  source 
selections.  Projection  bases  operate  tactical  and  strategic  aircraft,  i.e.,  fighters  and  bombers; 
airlifters;  and  space  assets.  Guard  and  reserve  units  are  included. 

The  other  base  category  includes  organizations,  units,  and  locations  that  either  do  not  have 
a  wartime  mission,  even  though  they  might  provide  essential  support  to  the  wartime  mission, 
or  typically  do  not  deploy  assets.  These  units  include  AETC,  the  human  systems  wing,  and  all 
direct  reporting  units  that  report  directly  to  the  Chief  of  Staff  of  the  Air  Force,  such  as  the  Air 
Force  Academy  and  Andrews  AFB. 

Two  points  are  worth  noting.  First,  product  centers  account  for  about  half  of  the  Air 
Force  procurement  spend  over  this  period.  So  it  is  not  surprising  that  Air  Force-wide  out¬ 
comes  are  similar  to  those  in  the  product  centers.  Second,  in  general,  only  the  outcomes  for 
technical  activities  and  other  base  activities  stand  out.  Experience  in  technical  activities  is  rela¬ 
tively  good;  that  in  other  base  activities  is  relatively  bad.  We  do  not  understand  the  reasons 
for  this  variation.  It  probably  warrants  further  investigation.  Product  centers  have  markedly 
higher  sustained  protest  rates  than  the  rest  of  the  Air  Force.  But  because  sustained  protest  rates 
remain  small  relative  to  all  corrective  actions,  this  distinction  does  not  lead  to  large  differences 
between  product  centers  and  the  rest  of  the  Air  Force.14 


13  Put  another  way,  the  primary  mission  of  these  bases  is  to  project  force. 
For  more  detail,  see  Light  et  al.,  2011,  Table  2.6. 


14 
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Air  Force  Share  of  DoD  Protests  Has  Been 


Declining  over  Last  Two  Decades 


Note:  Based  on  root  B-numbers  and  calculated  from  PACTS  and  GAO  data  for  FY  2000-08. 

RAND 
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We  do  not  have  the  degree  of  detail  on  non-Air  Force  agencies  that  PACTS  provides  on 
the  Air  Force.  But  by  combining  PACTS  and  GAO  data,  we  can  provide  a  high-level  com¬ 
parison  between  the  bid  protest  experience  of  the  Air  Force  and  the  experience  of  DoD  as  a 
whole.15  This  slide  shows  the  total  number  of  Air  Force  protests  as  a  share  of  the  total  number 
of  DoD  protests.  This  share  varies  significantly  from  year  to  year  but  displays  a  long-term  drop 
over  time.16  This  drop  occurs  because,  as  the  number  of  protests  has  fallen  for  both  the  Air 
Force  and  DoD,  the  number  of  protests  has  fallen  faster  in  the  Air  Force  (about  70  percent) 
than  in  DoD  as  a  whole  (about  65  percent)  over  the  period  shown.  Many  factors  could  account 
for  this  difference;  we  lack  the  detail  required  to  look  into  the  DoD  trend  and  have  focused  our 
attention  in  our  analysis  on  understanding  the  Air  Force  experience. 


15  For  the  numerical  data  underlying  this  figure,  see  Light  et  ah,  2011,  Table  2.7. 

16  The  straight  line  shown  is  a  simple  regression  line  for  the  points  in  the  graph. 
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Patterns  for  Contracts  and  Contract 
Awards  Have  Changed  over  15  Years 


Fiscal  Year 


RAND 


Source:  DD-350  and  FPDS-NG 
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Broader  Patterns  That  Might  Affect  Bid  Protest  Experience 

This  slide  provides  some  context  for  those  concerned  that  the  general  environment  in  which 
the  Air  Force  awards  contracts  might  explain  some  of  the  patterns  described  above.17  Perhaps 
the  most  important  trends  to  be  aware  of  are  those  for  total  number  of  contract  awards,  shown 
in  red,  and  total  number  of  competitive  awards,  shown  in  yellow.  These  two  numbers  track  one 
another  closely  through  FY  1994-2008.  The  number  of  awards  fell  through  the  1990s  as  the 
Air  Force  drew  down  its  force  structure  and  procurement  of  new  systems  and  then  rose  as  the 
Air  Force  relied  increasingly  on  external  providers  of  services  rather  than  on  more  traditional 
organic  sources  of  services.  Because  the  trends  for  total  and  competitive  contract  awards  move 
together  so  closely,  using  the  number  of  competitive  awards  to  normalize  number  of  protests 
and  protest  outcomes  on  earlier  slides  would  not  yield  results  that  differ  qualitatively  from 
those  based  on  total  number  of  awards.  The  number  of  C-  and  D-type  contract  awards,  which 
tend  to  be  larger  and  more  complex  than  other  contract  award  types,  fell  dramatically  through 
the  1990s  and  then  stabilized  following  FY  2000. 18  They  are  shown  in  green.  Despite  all  this 


17  This  graph  displays  numbers  contracted  from  data  in  DD350  and  FPDS-NG.  For  more  detail,  see  the  discussion  in  Light 
et  al.,  2011,  Figure  2.2. 

18  The  Federal  Acquisition  Regulation  (FAR)  “is  the  primary  regulation  for  use  by  all  federal  executive  agencies  in  their 
acquisition  of  supplies  and  services  with  appropriated  funds.”  (Quoted  in  the  “Foreword”  of  U.S.  General  Services  Admin¬ 
istration,  Department  of  Defense,  and  National  Aeronautics  and  Space  Administration,  Federal  Acquisition  Regulation, 
Washington,  D.C.,  March  2005.)  It  defines  many  types  of  contracts.  C-  and  D-type  contracts  are  indefinite  delivery  con¬ 
tracts  (C-type  contracts  state  a  definite  quantity;  D-type  contracts  do  not  state  a  definite  quantity)  that  tend  to  be  larger  and 
more  complex  than  other  contract  types  used  by  the  Air  Force. 
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turbulence,  the  total  number  of  contracts,  shown  in  blue,  changed  much  less.19  This  number 
was  relatively  stable  through  the  1990s  before  beginning  a  steady  rise  through  the  2000s. 

Taken  together,  these  trends  tell  us  that  both  the  level  and  the  pattern  of  Air  Force  con¬ 
tracting  behavior  changed  dramatically  over  this  period  of  time.  It  would  be  desirable  if  more 
detailed  statistical  analysis  of  Air  Force  experience  with  bid  protests  could  control  for  these 
patterns  of  change.  The  econometric  models  discussed  below  attempt  to  do  that. 


19  The  number  of  contracts  shown  in  any  year  is  the  number  of  distinct  contracts  in  which  transactions  occurred  in  that 
year.  The  number  of  contract  awards  shown  in  a  year  is  the  number  of  new  contracts  initiated  in  that  year.  So  the  number 
of  contracts  reflects  the  number  of  new  awards  as  well  as  the  number  of  contracts  initiated  in  the  past  and  still  active. 


Patterns  for  Bid  Protests  During  FY  1991-2008  41 


Other  Patterns  Point  to  a  Likely 
Rising  USAF  Acquisition  Workload 


RAND  Source.  DD-350  and  FPDS-NG  GAO  Bid  Protests  in  Air  Force  Source  Selections  -19- 


The  changes  documented  on  the  last  slide  in  all  likelihood  brought  changes  in  workload 
for  the  acquisition  workforce.  This  slide  displays  two  numbers  that  provide  some  insight  into 
such  changes.20  The  level  of  spending  on  procurement,  shown  in  constant  2000  dollars  in 
red,  is  keyed  to  the  left  vertical  axis.  It  shows  relative  stability  through  the  1990s,  despite  the 
changes  afoot  then.  Expenditure  rose  rapidly  through  the  early  2000s  before  stabilizing  again 
at  a  much  higher  level.  At  the  same  time,  the  number  of  transactions,  shown  in  blue  and  keyed 
to  the  right  vertical  axis,  also  changed  but  less  dramatically.21  A  slow  descent  through  the  mid- 
1990s  led  to  a  recovery  by  about  2000  and  a  continuing  rise  through  the  period. 

If  we  posit  a  positive  relationship  between  each  of  these  and  workload,  we  would  expect 
increasing  demands  on  the  workforce  over  the  last  decade.  That  said,  improved  management 
of  acquisition  information  systems  and  dramatic  changes  in  contracting  associated  with  sim¬ 
plified  acquisition  and  other  acquisition  reforms  might  well  have  offset  the  potential  effects  of 
these  trends  on  workload.  Changes  in  the  level  and  composition  of  the  acquisition  workforce 
would  also  obviously  shape  the  effects  of  such  trends.  Unfortunately,  without  information  on 
the  contractor  workforce  that  supported  the  organic  Air  Force  acquisition  workforce  through 
this  period,  it  is  difficult  to  control  for  changes  in  total  workforce  relevant  to  any  changes  in 
acquisition  workload.22 


20  This  graph  displays  numbers  contracted  from  data  in  DD350  and  FPDS-NG.  Note  that  the  number  of  actions  is  differ¬ 
ent  from  the  number  of  contracts  or  number  of  awards  shown  on  the  previous  slide.  For  more  detail,  see  the  discussion  in 
Light  et  al.,  2011,  Figure  2.4. 

21  A  “transaction”  is  the  unit  of  observation  in  DD350  and  FPDS-NG  and  effectively  signals  a  significant  contracting 
action  associated  with  a  change  in  the  level  of  Air  Force  funds  obligated. 

For  information  on  trends  in  the  organic  workforce  over  some  of  this  period,  see  Kayes  2008,  Charts  29-30. 


22 
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“All  Else  Equal,”  Several  Factors  Affect 
Total  Number  of  Protests 

•  Year  to  year,  total  number  of  protests  has  fallen  8.7 
to  9.7%  a  year 

*  Within  any  year,  total  number  of  protests  has  risen  as 
number  of  awards  rose  but  not  proportionally 

•  Total  number  of  protests  has  fallen  more  than 
proportionally  with  share  of  competitive  awards 

*  Total  number  of  protests  has  varied  across  different 
types  of  contracting  centers 

-About  the  same  in  headquarters,  product  centers, 
and  air  logistics  centers  (with  80%) 

-About  45%  lower  in  technical  activities  (5%) 

-About  75%  higher  in  other  locations  (14%) 

Note:  Based  on  root  B-numbers.  Results  robust  across  functional  forms  for  1994-2008, 
n  =  90.  All  results  significant  at  a  =  .05  or  better.  Data  from  PACTS,  DD-350,  FPDS-NG. 
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Isolating  Factors  That  Affect  Bid  Protest  Experience 

Methods  are  available  to  test  statistically  what  factors  have  affected  the  number  of  protests  the 
Air  Force  has  experienced  in  various  circumstances.  These  methods  in  effect  use  historical  data 
on  Air  Force  experience  to  vary  one  factor  while  holding  all  others  constant  and  test  to  see 
whether  this  change  affects  the  number  of  protests  experienced  while  “all  else  is  equal.” 

Factors  Affecting  the  Total  Number  of  Protests 

We  used  a  family  of  regression-based  models  to  do  this  with  data  on  Air  Force  experience  from 
the  period  FY  1994-2008. 23  These  models  sought  to  explain  the  number  of  protests  observed 
in  one  year  at  one  type  of  contracting  center  in  terms  oP4 

•  the  number  of  contract  awards  in  this  year  and  type  of  contracting  center 

•  the  number  of  competitive  contract  awards  in  this  year  and  type  of  contracting  center 

•  the  share  of  these  awards  that  was  competitive 

•  the  share  of  these  awards  with  C-  and  D-type  contracts 

•  the  average  Air  Force  expenditure  per  contract  in  this  year  and  type  of  contracting  center 


23  Alternative  members  of  the  family  of  regressions  treat  the  total  number  of  protests  as  being  distributed  normally,  nega¬ 
tive  binomially,  or  Poisson  and  hold  different  sets  of  factors  constant  while  testing  for  the  effects  of  each  one.  For  details  on 
the  methods  used,  see  Light  et  ah,  2011. 

24  Some  have  argued  that  factors  outside  the  Air  Force  acquisition  system,  such  as  the  general  state  of  the  economy,  might 
affect  the  trends  examined  here.  Our  analysis  considers  only  factors  that  can  be  directly  linked  to  Air  Force  acquisitions. 
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•  the  type  of  contracting  center 

•  the  year. 

Different  models  used  different  combinations  of  these  factors.  We  expected  to  observe  the 
following  effects.  All  else  equal: 

•  The  number  of  protests  would  rise  proportionally  with  number  of  contract  awards  or 
number  of  competitive  contract  awards,  suggesting  that  the  probability  of  experiencing 
the  protest  of  a  contract  award  is  independent  of  the  number  of  awards. 

•  The  number  of  protests  would  rise  with  the  share  of  competitive  contract  awards,  holding 
total  number  of  contract  awards  constant,  because  there  are  more  offerors  and  so  pre¬ 
sumably  more  opportunities  for  perceived  errors  in  competitive  than  in  noncompetitive 
awards. 

•  The  number  of  protests  would  rise  with  the  share  of  C-  and  D-type  contracts,  because 
protests  would  be  more  likely  in  larger  awards  with  more  complex  contracts. 

•  The  number  of  protests  would  rise  with  the  average  Air  Force  expenditure  per  contract, 
because  protests  would  be  more  likely  where  the  Air  Force  spends  more  per  contract, 
increasing  the  stakes  and  perhaps  the  complexity  in  its  contracts.25 

•  The  number  of  protests  would  be  higher  in  types  of  contracting  centers  with  larger  and 
more  complex  contracts — for  example,  higher  in  headquarters  and  product  centers  than 
in  projection  and  other  bases. 

•  The  number  of  protests  would  fall  over  time,  reflecting  the  strong  relationship  identified 
above  when  observing  trends  over  time. 

We  had  hoped  to  include  information  about  personnel  in  the  acquisition  workforce.  As 
noted  above,  without  data  on  contractor  personnel  who  supported  the  Air  Force  acquisition 
workforce,  we  could  not  construct  meaningful  measures  of  the  numbers,  types,  or  skills  of 
personnel  in  the  total  Air  Force  acquisition  workforce  in  different  years  and  types  of  contract 
centers. 

The  slide  summarizes  the  statistically  significant  results  that  we  observed  robustly  across 
alternative  models.26  All  else  equal: 

•  From  year  to  year,  the  total  number  of  protests  has  fallen  at  a  remarkable  8  to  9  percent 
per  year,  even  faster  than  implied  by  the  numbers  on  Slide  14. 

•  Within  any  year,  the  total  number  of  protests  has  risen  as  the  number  of  contract  awards 
rose  but  less  than  proportionally.  It  is  hard  to  understand  what  causes  the  coefficient 
to  differ  significantly  from  one,  the  value  consistent  with  total  proportionality,  but  the 
results  are  robust  across  functional  forms.27 


25  This  bullet  and  the  one  above  use  two  different  independent  variables  to  detect  a  similar  effect. 

26  The  tests  used  found  at  least  a  95  percent  probability  that  an  effect  reported  as  “significant”  was  different  from  zero.  For 
most  of  the  significant  effects  reported  here,  the  probability  was  99  percent.  These  results  are  quite  high  for  models  with 
only  87  to  90  observations  each  and  less-than-perfect  data. 

27  Here  are  two  possibilities.  First,  suppose  that  the  Air  Force  buys  two  kinds  of  things.  For  the  first,  the  number  of  con¬ 
tracts  it  writes  is  constant  from  year  to  year.  For  the  second,  the  number  of  contracts  it  writes  goes  up  and  down.  And  sup¬ 
pose  that  the  Air  Force  experiences  a  higher  rate  of  protests  for  the  first  kind  than  for  the  second.  If  that  were  true,  we  would 
observe  the  pattern  that  the  statistical  analysis  identifies.  That  said,  one  might  reasonably  expect  that  the  rate  of  protests 
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•  Within  any  year,  the  total  number  of  protests  has  also  risen  as  the  number  of  competitive 
contract  awards  rose,  but  less  than  proportionally.  The  relationship  here  is  closer  to  pro¬ 
portional  than  for  total  contract  awards,  but  the  value  of  the  coefficient  remains  signifi¬ 
cantly  below  one.  This  implies  some  mechanism  similar  to  that  described  above. 

•  Holding  the  total  number  of  contract  awards  constant,  the  number  of  protests  fell  more 
than  proportionately  when  the  share  of  competitive  contract  awards  rose,  suggesting  that 
competition  has  made  protests  less  likely  rather  than  more  likely,  as  we  had  expected.  This 
result  is  less  robust  and  less  significant  than  the  other  results  reported  here. 

•  Headquarters,  product  centers,  and  ALCs,  defined  as  they  were  above,  have  similar  expe¬ 
rience  with  protests.  These  account  for  the  lion’s  share  of  Air  Force  procurement.  Tech¬ 
nical  centers,  with  about  5  percent  of  the  total  spend,  have  substantially  better  experi¬ 
ence,  with  45  percent  fewer  protests  than  normal  for  the  Air  Force.  They  are  likely  to 
have  qualitatively  different  types  of  contracts  and  may  sustain  a  different  kind  of  culture 
within  the  Air  Force.  As  noted  above,  there  are  probably  useful  lessons  to  be  learned  in 
technical  contracting  centers  for  the  rest  of  the  Air  Force.  We  did  not  attempt  to  identify 
them  during  this  study.  Projection  and  other  bases,  with  24  percent  of  the  spend,  have 
substantially  worse  experience,  with  75  percent  more  protests  than  normal  for  the  Air 
Force.  The  operational  contracts  that  predominate  here  are  likely  simpler,  but  the  skill 
level  of  the  acquisition  workforce  is  also  likely  lower. 

•  Further  efforts  to  identify  the  effects  of  larger  or  more  complex  acquisitions  with  expendi¬ 
ture  per  contract  and  share  of  C-  and  D-type  contracts  found  no  evidence  to  suggest  that 
these  factors  affected  the  number  of  protests  experienced. 


Factors  Affecting  the  Number  of  Substantive  Protests 

Efforts  to  identify  the  factors  that  affect  the  number  of  substantive  protests — protests  that  led 
to  corrective  actions  or  merit  reviews — yielded  qualitatively  very  similar  findings.  These  efforts 
applied  the  same  models.  We  simply  changed  the  relevant  dependent  variable  for  each  regres¬ 
sion  tested.  Here  are  the  statistically  significant  results  that  we  observed  robustly  across  alterna¬ 
tive  models,  with  all  else  being  equal: 

•  From  year  to  year,  the  total  number  of  substantive  protests  has  fallen  6  to  8  percent  per 
year.  This  is  presumably  somewhat  slower  than  for  the  total  number  of  protests  because 
the  number  of  protests  leading  to  dismissals  and  withdrawals  without  corrective  action 
fell  so  much  over  the  1990s.  But  the  fall  in  substantive  protests  has  still  been  dramatic. 


would  be  lower  for  products  that  are  regularly  purchased  in  fixed  quantities  (since  one  would  expect  an  established  acquisi¬ 
tion  process  to  develop  for  such  products)  than  for  products  for  which  acquisitions  are  highly  variable  from  year  to  year.  In 
any  case,  we  cannot  identify  any  attributes  of  things  that  the  Air  Force  buys  that  would  induce  this  to  occur. 

Second,  suppose  that,  because  GAO  has  fixed  resources  for  investigating  protests,  it  can  investigate  only  a  fixed  number  of 
protests  each  year.  If  the  number  of  contract  awards  goes  up  in  a  year,  the  number  of  protests  will  also  rise,  but  the  number 
of  merit  reviews  will  not.  As  a  result,  the  likelihood  of  a  successful  protest  will  fall,  discouraging  protests  in  the  first  place 
and  so  leading  to  a  less-than-proportional  rise  in  protests  as  the  number  of  contract  awards  rises.  We  have  no  empirical 
evidence  of  such  inflexibility  at  GAO  and  rapid  learning  among  potential  protesters.  We  have  heard  no  mention  of  such 
inflexibility  or  learning  in  interviews.  But  a  closer  look  might  detect  them  both.  We  thank  Paul  Heaton  for  bringing  this 
second  possibility  to  our  attention. 
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Within  any  year,  the  total  number  of  substantive  protests  has  risen  as  the  number  of 
contract  awards  rose,  but  less  than  proportionally.  The  number  of  substantive  protests  is 
even  further  from  proportionality  than  the  total  number  of  protests.  But  the  distinction 
observed  between  total  number  of  contract  awards  and  number  of  competitive  contract 
awards  does  not  occur  for  substantive  protests. 

Holding  the  total  number  of  contract  awards  constant,  the  share  of  competitive  contract 
awards  has  no  effect  on  the  number  of  substantive  protests.  This  is  consistent  with  the 
result  reported  immediately  above. 

Headquarters,  product  centers,  and  ALCs,  defined  as  they  were  above,  have  similar  expe¬ 
rience  with  the  number  of  substantive  protests.  Technical  centers  again  have  substantially 
better  experience,  with  60  percent  fewer  protests  than  normal  for  the  Air  Force.  Projec¬ 
tion  and  other  bases  again  have  substantially  worse  experience,  with  70  percent  more 
protests  than  normal  for  the  Air  Force. 

Further  efforts  to  identify  the  effects  of  larger  or  more  complex  acquisitions  with  expendi¬ 
ture  per  contract  and  share  of  C-  and  D-type  contracts  found  no  evidence  to  suggest  that 
these  factors  affect  the  number  of  substantive  protests  experienced. 
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“All  Else  Equal,”  Several  Factors  Affect  the 
Share  of  Corrective  Actions  in  Total  Protests 

•  Corrective  actions  have  become  more  likely  over 
time  (sustained  protests  have  not) 

•  Poorly  crafted  RFPs  and  faulty  treatment  of  offerors 
have  been  more  likely  to  result  in  corrective  actions 
than  other  errors  in  source  selection 

•  Corrective  actions  have  occurred  more  often  in 
acquisitions  with  more  B-numbers 

•  Likelihood  of  corrective  action  has  not  differed 
much  across  types  of  contracting  centers  or 
products  acquired 

Note:  Results  based  on  logistic  regression  of  PACTS  data  on  root  B-numbers 
for  1991-2008,  n  =  2482.  All  significant  results  have  a  =  .05  or  better. 

RAND 
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Factors  Affecting  the  Likelihood  of  Corrective  Actions  in  Response  to  Protests 

Another  way  to  test  whether  various  factors  have  affected  Air  Force  experience  with  bid  pro¬ 
tests  is  to  ask  what  factors  affect  the  likelihood  of  some  protest  outcomes  when  a  protest  occurs. 
To  do  this,  we  used  logistics  regression,  applied  as  an  analysis  of  covariance  to  a  dependent 
variable  with  a  logistic  distribution.28  Because  sustained  protests  and  corrective  actions  appear 
to  impose  the  largest  costs  on  the  Air  Force  when  it  experiences  a  protest,  we  applied  this  analy¬ 
sis  in  models  that  used  as  a  dependent  variable,  alternatively 

1.  occurrence  of  a  sustained  protest 

2.  occurrence  of  a  corrective  action 

3.  occurrence  of  either  a  sustained  protest  or  a  corrective  action. 

In  this  analysis  of  covariance,  we  tested  whether  the  following  factors  affected  the  like¬ 
lihood  that  one  of  these  outcomes  occurred:  (1)  date,  (2)  type  of  commodity  bought  in  the 
acquisition,  (3)  stated  basis  for  the  protest,  (4)  type  of  contract  center,  and  (5)  number  of 


28  The  results  are  based  on  2,482  total  protests  and  1,059  substantial  protests  that  occurred  during  FY  1991-2008.  For 
detailed  quantitative  findings  and  a  description  of  the  details  of  how  we  did  this,  see  Light  et  al.,  2011. 
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B-numbers  in  the  acquisition.29  Using  this  method  to  “hold  all  else  equal,”  we  developed  rather 
limited  results,  including  the  following.30 

When  a  protest  occurred,  a  corrective  action  became  steadily  more  likely  as  time  passed. 
This  effect  was  strongest  in  the  1990s.  When  starting  with  a  substantive  protest,  the  strong 
effect  of  time  on  corrective  actions  disappears.  This  suggests  that,  when  a  protest  occurred,  the 
likelihood  of  a  substantive  protest  increased  over  time.  This  interpretation  is  consistent  with 
the  observation  above  that  much  of  the  improvement  in  the  Air  Force’s  experience  with  bid 
protests,  especially  during  the  1990s,  was  the  product  of  more  knowledgeable  protesters  who 
filed  fewer  protests  that  would  be  dismissed  or  withdrawn  without  corrective  action. 

Given  either  a  protest  or  a  substantive  protest,  the  likelihood  of  a  sustained  protest  did  not 
change  over  time.  This  may  be  because  sustained  protest  patterns  have  been  too  sporadic  (and 
unimportant  relative  to  corrective  actions)  to  define  comparable  trends  for  them. 

The  basis  for  protest  affected  the  likelihood  of  a  corrective  action,  but  identifying  the 
exact  effect  is  challenging.  Given  a  protest,  claims  of  a  faulty  RFP  or  faulty  treatment  of  an 
offeror  appear  somewhat  more  likely  than  other  claims  to  have  led  to  a  corrective  action.  Given 
a  substantive  protest,  only  the  claim  of  a  faulty  RFP  stands  out  as  exceptional  in  its  effect  on 
the  likelihood  of  a  corrective  action. 

The  effect  of  the  basis  of  protest  on  the  likelihood  of  a  sustained  protest  is  even  murkier. 
Given  a  protest,  a  sustained  protest  was  somewhat  more  likely  when  the  basis  of  protest  involved 
a  faulty  sourcing  decision.  In  a  faulty  sourcing  decision,  the  Air  Force  typically  inappropri¬ 
ately  excludes  an  offeror  from  a  source  selection  or  does  not  recognize  the  special  status — for 
example,  as  a  small  business — that  an  offeror  claims  in  a  source  selection.  Given  a  substan¬ 
tive  protest,  any  such  effect  disappears.  This  suggests  that  a  faulty  sourcing  decision  may  have 
increased  the  likelihood  of  a  substantive  protest;  stated  differently,  protesters  have  been  least 
likely  to  hie  a  protest  in  error  when  they  have  hied  a  claim  of  a  faulty  sourcing  decision. 

The  likelihood  of  observing  a  corrective  action  rose  as  the  number  of  B-numbers  associ¬ 
ated  with  the  acquisition  protested  rose.  We  do  not  know  why  this  occurred.  Consider  three 
possibilities: 

•  Suppose  the  number  of  B-numbers  is  completely  exogenous  to  the  outcome  of  a  protest. 
That  is,  GAO  chooses  a  number  of  dockets  in  a  way  that  is  unrelated  to  the  likelihood 
that  any  of  them  will  precipitate  corrective  action  or  lead  to  a  sustained  protest.  If  this 
occurred,  GAO  may  have  created  more  dockets  to  administer  potentially  more  complex 
protests.  If  more  complex  protests  posed  higher  risk  to  the  Air  Force,  the  correlation 
detected  in  our  analysis  may  simply  reflect  an  Air  Force  tendency  to  be  more  conservative 
in  more  complex  protests. 

•  An  exogenous  number  of  protest  grounds  reflected  in  the  multiple  B-numbers  in  a  single 
procurement  could  create  the  results  detected  in  another  way.  Each  filing  of  additional 
protest  grounds  may  have  given  protesters  an  additional  opportunity  to  get  a  sustainment. 


29  The  analysis  considered  the  following  categories  for  each  factor:  (1)  Date:  two-year  periods  from  1991  to  2008.  (2)  Type 
of  commodity:  weapon,  supply,  construction,  research  and  development,  other  service  (based  on  commodity  type  identified 
in  PACTS).  (3)  Basis  of  protest:  faulty  sourcing  decision,  faulty  RFP,  faulty  evaluation,  faulty  treatment  of  offeror,  other 
(based  on  basis  of  protest  identified  in  PACTS).  (4):  Type  of  contracting  center:  headquarters,  product  center,  air  logistics 
center,  technical  activity,  projection  base,  other  base  (defined  as  above). 

30  As  noted  above,  we  believe  good  data  on  the  values  of  procurements  and  the  reasons  for  voluntary  corrective  actions 
would  have  helped  us  generate  better  and  more  helpful  results.  Such  data  are  not  easily  accessible. 
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If  the  GAO  review  process  is  somewhat  stochastic,  adding  more  protest  grounds  by  addi¬ 
tional  filings  might  be  seen  as  adding  opportunities  for  GAO  to  rule  in  the  protesters’ 
favor  on  at  least  one  of  them.  Knowing  this,  the  Air  Force  could  have  preempted  GAO 
merit  review  by  volunteering  corrective  action  more  often  as  the  number  of  protest  filings 
rose. 

•  Suppose,  on  the  other  hand,  that  the  number  of  B -numbers  reflects,  at  least  to  some 
degree,  GAO’s  interest  in  a  protest,  an  interest  that  presumably  rises  as  the  likelihood 
of  a  merit  review  rises.  In  this  case,  the  Air  Force  might  have  taken  evidence  of  more 
B-numbers  as  evidence  of  higher  GAO  interest  and,  potentially,  higher  likelihood  of  sig¬ 
nificant  costs  or  even  a  bad  outcome  in  a  merit  review.  In  response,  the  Air  Force  might 
have  responded  to  B-numbers  by  becoming  more  willing  to  offer  corrective  action. 

These  alternatives  are  not  mutually  exclusive.  The  highly  significant  correlation  detected 
suggests  that  these  deserve  further  attention. 

Given  either  a  protest  or  a  substantive  protest,  the  type  of  commodity  bought  and  type 
of  contracting  center  had  no  discernible  effect  on  the  likelihood  of  a  sustained  protest  or  a  cor¬ 
rective  action.  Types  of  contracting  centers  are  relatively  well  defined  in  our  database;  types  of 
products  acquired  are  not.  So  we  have  more  confidence  in  the  results  for  contracting  centers 
than  in  those  for  products  acquired. 

The  negative  results  for  type  of  commodity  and  the  murky  results  for  basis  of  protest 
may  reflect  the  quality  of  data  in  PACTS.  Our  review  of  the  quality  of  PACTS  data  on  both 
of  these  factors  detected  serious  inconsistencies  and  irregularities  that  forced  us  to  scrub  the 
data  thoroughly  before  application.  For  example,  ways  of  defining  bases  of  protest  appear  to 
have  changed  over  time.  And  it  is  unclear  how  the  person  providing  data  to  PACTS  decided 
which  of  a  list  of  bases  of  protest  to  use  to  label  any  one  protest.  Similar  concerns  apply  to  the 
data  on  type  of  commodity.  These  concerns  point  to  likely  data  errors  that  reduce  the  likeli¬ 
hood  that  coefficients  based  on  such  data  would  be  significant.  We  report  these  findings  with 
caution  and  suggest  that  the  effects  of  basis  of  protest  and  type  of  protest  would  benefit  from 
additional  attention. 

Taken  together,  the  results  reported  in  this  chapter  sustain  the  positive  picture  developed 
in  the  first  chapter  of  the  Air  Force’s  broad  experience  with  GAO  bid  protests.  The  next  chapter 
turns  to  a  detailed  analysis  of  two  Air  Force  acquisitions  with  much  less  satisfactory  outcomes. 
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Protests  Sustained  in  the  CSAR-X  and  KC-X  Source  Selections 
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The  CSAR-X  and  KC-X  source  selections  both  experienced  sustained  protests.  These  protests 
led  to  considerable  negative  publicity  and  seriously  set  back  high-priority  Air  Force  acquisi¬ 
tion  programs.  Yet  despite  their  perceived  importance,  we  found  no  formal,  publicly  available 
comprehensive  documentation  published  in  a  single  place  of  what  happened  in  these  source 
selections.1  The  Air  Force  asked  RAND  to  document  both.  In  this  chapter,  we  focus  on  the 
sustained  protests  and  lessons  that  can  be  learned  from  them. 

This  chapter  does  not  offer  complete  case  histories  of  these  two  episodes.  The  source 
selections  and  protests  for  these  two  programs  took  place  against  a  background  of  a  wide 
variety  of  extremely  complex  and  interrelated  political,  economic,  military,  bureaucratic,  and 
other  factors.  A  full  accounting  of  what  happened  on  each  of  these  programs  would  require  a 


1  The  Air  Force  and  the  contractors,  of  course,  retain  extensive  internal  documentation  on  these  cases,  but  this  material  is 
source-selection-sensitive  and  proprietary  and  not  available  to  the  public.  GAO  published  summaries  of  its  specific  decisions 
with  all  source  selection  and  proprietary  data  excised.  However,  these  summaries  do  not  provide  full  and  comprehensive 
accounts  that  place  the  programs  in  a  wider  program  acquisition  history  and  context.  There  is,  of  course,  a  vast  amount 
of  documentation  available  in  numerous  articles  in  the  aerospace  trade  press,  but  most  of  this  material  focuses  on  specific 
news  events  as  the  story  of  the  protests  unfolded.  None  of  them  provide  a  comprehensive  overview  of  the  entire  history  of 
the  protests  within  the  context  of  the  larger  acquisition  program  histories. 
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book-length  scholarly  treatment.  We  focus  on  what  we  believe  will  be  of  greatest  immediate 
importance  to  the  Air  Force — a  thorough  overview  of  the  specific  sustained  protests  on  these 
programs,  some  background  context,  and  our  analysis  of  the  key  lessons  the  Air  Force  should 
take  away  from  these  specific  experiences. 
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Sustained  CSAR-X  Protests  Added  Delays  to 
Long-Delayed  Helicopter  Replacement 

*  2001 :  AoA  compares  capabilities  of  existing  candidates  for  new 
CSAR  helicopter.  USAF  decides  to  buy  132  medium-lift  aircraft. 

*  2005:  CSAR-X  acquisition  begins. 

*  Nov  2006:  Boeing  wins.  Lockheed-Martin  (L-M)  and  Sikorsky 
protest. 

*  Feb  2007:  GAO  sustains  one  protest  (#1)  on  USAF  evaluation  of 
cost. 

*  Mar  2007:  USAF  requests  clarification;  GAO  denies  all 
remaining  merit  protests. 

•May  2007:  USAF  issues  revised  RFP.  L-M  protests.  Sikorsky 
joins  protest. 

*  Aug  2007:  GAO  sustains  protest  (#2)  on  USAF  failure  to  take 
GAO’s  suggested  corrective  action. 

*  Jun  2009:  USAF  cancels  CSAR-X  program. 

RAND 
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Combat  Search  and  Rescue  Helicopter  Program 

CSAR-X  experienced  two  sustained  protests,  the  second  the  direct  consequence  of  the  Air 
Force’s  failure  to  take  appropriate  corrective  action  to  fully  address  the  first.  Each  was  technical 
in  character  but  readily  predictable  if  viewed  from  the  right  perspective. 

These  protests  initially  led  to  further  delay  in  an  already  much  delayed  acquisition.  In 
itself,  this  additional  delay  was  small.  But  the  first  protest  left  the  program  office  exhausted, 
depleted,  and  poorly  prepared  to  respond  to  the  second  protest.  Persistent  delay  ultimately  left 
the  program  wounded  and  vulnerable  to  ongoing  events  elsewhere  in  DoD.  When  these  events 
overtook  the  program,  the  Secretary  of  Defense  asked  the  Air  Force  to  redirect  its  priorities 
elsewhere,  leaving  the  program  effectively  in  limbo. 

The  CSAR-X  program  began  as  the  Personnel  Recovery  Vehicle  (PRV)  program  in  the 
late  1990s.  Between  1981  and  1997,  the  Air  Force  procured  112  Sikorsky  FIH-60G  helicopters 
to  fulfill  the  personnel  recovery  and  combat  search  and  rescue  (CSAR)  missions.  By  the  end  of 
the  1990s,  the  Air  Force  began  serious  consideration  of  a  replacement  program  for  the  HH-60 
because  of  a  variety  of  factors,  including  mission  performance  shortcomings  of  the  HH-60, 
declining  numbers  because  of  accidents  and  other  losses,  and  the  rapidly  approaching  end  of 
the  useful  mission  design  life  for  the  oldest  HH-60s.  In  January  1999,  ACC  developed  a  Mis¬ 
sion  Need  Statement  (MNS)  for  a  new  PRV.2  Initially  there  was  widespread  interest  within  the 
Air  Force  in  procuring  the  Bell-Boeing  CV-22  Osprey  tilt-rotor  aircraft  to  fulfil  this  mission, 
but  ACC  planned  to  conduct  an  AoA  once  the  MNS  was  approved  to  determine  the  most 


2 


New  Helicopters  Possibly  in  Store  for  Enhanced  Combat  Search  and  Rescue,”  Inside  the  Air  Force,  February  12,  1999. 
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cost-effective  solution.3  The  program  remained  in  flux  for  many  months  as  a  variety  of  differ¬ 
ent  options  were  considered,  including  combining  the  PRV  requirement  with  a  requirement 
for  replacing  the  Presidential  Helicopter  (VXX  Program),  as  well  as  all  other  U.S.  Air  Force 
helicopters.  Because  of  the  lack  of  budgetary  resources  and  discussions  over  reorganization  of 
CSAR  assets  the  program  moved  forward  slowly. 

The  Air  Force  completed  its  formal  AoA  in  mid-2001,  concluding  that  a  larger  number 
(at  that  time  put  at  132  aircraft,  later  changed  to  141)  of  more  capable  medium-lift  helicopters 
would  be  needed  to  meet  the  future  PRV  and  CSAR  missions.  By  this  time,  the  CV-22  was 
increasingly  considered  too  expensive  for  the  larger  number  of  platforms  required.4  Given  bud¬ 
getary  restrictions,  planners  began  focusing  more  on  existing,  previously  developed  platforms. 
Extensive  market  research  was  completed  in  August  2004.  The  AoA  and  market  research  sug¬ 
gested  that  existing  platforms  could  serve  as  good  baselines  for  the  system,  but  significant 
modifications  would  be  necessary. 

Possible  candidates  examined  in  the  AoA  and  market  research  included  the  Westland- 
Augusta  EH-101,  the  Sikorsky  S-92,  the  European  Aeronautic  Defence  and  Space  Company 
(EADS)  Eurocopter  MH-90,  and  the  Sikorsky  H-60X  upgrade  proposal.  Thus,  it  was  thought 
likely  that  the  PRV/CSAR  missions  could  be  fulfilled  by  modifications  of  existing  platforms 
such  as  these,  two  of  which  had  been  developed  by  foreign  contractors.5 

Requirements  documentation  was  thoroughly  vetted  and  passed  through  the  new  JCIDS 
process.  The  MNS  was  validated  and  transitioned  into  a  formal  Operational  Requirements 
Document  (ORD),  and  validated  by  the  Joint  Requirements  Oversight  Council  (JROC)  in 
February  2004.  In  line  with  the  new  JCIDS  process  promulgated  by  the  Joint  Staff,  the  ORD 
was  converted  to  a  CDD  through  a  rigorous  procedure  carried  out  from  the  summer  of  2004 
through  June  2005. 

The  Air  Force  planned  to  begin  the  formal  acquisition  before  the  end  of  2005.  Initially, 
many  observers  believed  that  Sikorsky  had  the  inside  track  on  the  competition  with  its  com¬ 
mercially  derived  HH-92,  because  it  was  the  incumbent  contractor  (having  provided  the  exist¬ 
ing  PRV/CSAR  HH-60  aircraft),  and  because  the  US  101  was  a  foreign-derivative  helicopter. 
However,  in  January  2005,  the  Lockheed  Martin-led  EH  101  team  surprised  many  in  the 
defense  press  by  winning  the  VXX  Presidential  Helicopter  competition,  leading  many  observ¬ 
ers  to  speculate  that  the  much  larger  PRV/CSAR  contract  would  now  be  intensely  competitive.6 

This  was  indeed  correct,  in  that  a  variety  of  factors  during  2005  continued  to  delay  the 
beginning  of  the  acquisition  program  and  led  to  an  intensification  of  the  competition.  Con¬ 
gress  temporarily  cut  funding  for  the  PRV/CSAR  program  and  urged  DoD  to  speed  up  acqui¬ 
sition  of  a  new  helicopter.  This  could  be  accomplished  by  reducing  PRV/CSAR  program  costs 
through  a  variety  of  measures,  including  greater  focus  on  the  acquisition  of  an  off-the-shelf  or 


3  “CV-22  Likely  to  Take  Over  All  Current  Air  Force  Rotorcraft  Missions,”  Inside  the  Air  Force,  November  27,  1998. 

4  Increased  individual  platform  capability  could  not  be  fully  traded  off  without  constraint  against  reduced  procurement 
numbers.  This  was  because  the  AoA  analysis  demonstrated  that  a  specific  number  of  aircraft  were  needed  for  full  geographi¬ 
cal  coverage  and  tactical  and  operational  reasons.  See  Bethany  Scott,  “Air  Force  Eyes  Replacement  for  Aging  Pave  Hawk 
Helos,”  National  Defense  Magazine,  September  2001. 

5  In  October  2001,  Lockheed  Martin  announced  its  intention  to  enter  into  a  joint  agreement  with  Westland-Augusta  to 
develop  and  market  a  version  of  the  EH-101  (called  the  US  101)  for  the  U.S.  market. 

6  John  Liang,  “Wall  Street  Analysts  Surprised  by  Lockheed  Team’s  Presidential  Helicopter  Win,”  DefenseAlert-Daily 
News,  January  31,  2005. 


Protests  Sustained  in  the  CSAR-X  and  KC-X  Source  Selections  53 


low-risk  commercially  derived  aircraft  requiring  minimal  development.7  Congress  attempted 
to  impose  special  conditions  on  DoD  for  the  competition,  including  certified  independent 
cost  estimates  of  development  and  production,  as  well  as  the  requirement  to  fully  document 
projected  operating  and  support  (O&S)  costs.  These  conditions  were  included  in  the  House 
Armed  Services  Committee  program  authorizations,  which  appeared  in  May,  well  before  the 
Air  Force  issued  its  final  RFP,  thus  alerting  the  Air  Force  to  the  extreme  sensitivity  of  Congress 
to  cost  issues  and  the  importance  of  accurate  cost  estimates,  for  both  procurement  and  O&S.8 

With  no  other  major  rotary-wing  programs  on  the  horizon,  contractor  competition  for 
PRV/CSAR  greatly  intensified.  The  total  potential  program  value  was  now  estimated  by  indus¬ 
try  observers  to  exceed  $10  billion  to  $12  billion.  Not  surprisingly,  the  publicly  perceived  lead¬ 
ing  candidate  teams,  headed  by  Lockheed-Martin  with  its  US  101  team  and  Sikorsky  with  its 
HH-97,  redoubled  their  efforts.  The  competition  rose  to  an  even  higher  level  of  intensity  in  the 
summer  of  2005  when  Boeing  entered  the  fray  at  the  eleventh  hour  with  a  new  and  unantici¬ 
pated  competitive  design. 

In  August,  the  Air  Force  formally  changed  the  program  name  from  PRV  to  CSAR-X, 
reportedly  to  emphasize  the  narrower  scope  of  the  program  as  merely  an  off-the-shelf  helicop¬ 
ter  replacement  effort.  Congressional  actions  and  the  new  focus  on  mature  technology  and 
off-the-shelf  solutions,  which  was  ultimately  explicitly  mandated  by  the  Senate  version  of  the 
defense  appropriations  bill  passed  in  October,  effectively  guaranteed  the  elimination  of  the 
Bell/Boeing  CV-22  at  that  time  as  too  costly  and  high  risk. 

In  the  spirit  of  supporting  acquisition  of  a  lightly  modified  existing  platform,  the  Air 
Force  conducted  a  non-developmental  aircraft  flight  evaluation  of  the  leading  contenders 
during  the  summer.  It  was  at  this  time,  with  the  renewed  emphasis  on  off-the-shelf  solu¬ 
tions,  that  Boeing  surprised  the  existing  competitors  by  proposing  a  modification  of  veteran 
MH-47G  Chinook,  an  existing  platform  already  fully  militarized  with  many  of  the  subsystem 
capabilities  required  for  the  CSAR  and  already  in  use  with  the  U.S.  Special  Operations  Com¬ 
mand.  Consequently,  as  the  only  aircraft  already  in  the  U.S.  military  inventory  and  already 
militarized,  many  observers  felt  that  Boeing  with  its  HH-47  proposal  had  been  able  to  pull  off 
a  last  minute  coup  and  pull  ahead  of  the  other  two  long-time  leading  contenders  in  response 
to  the  changed  circumstances  driven  at  least  in  part  by  Congress.9 

The  Air  Force  issued  the  final  RFP  on  October  5,  2005,  with  submissions  due  by  Novem¬ 
ber  21.  One  year  later,  on  November  9,  2006,  the  Air  Force  awarded  the  CSAR-X  contract 
to  Boeing  for  its  HH-47  proposal.  This  was  a  Best  Value  Source  Selection.10  In  assessing  the 
four  basic  categories  of  evaluation  factors  (mission  capability,  proposal  risk,  past  performance, 


7  Martin  Matishak,  “Pentagon  Appeals  Host  of  CSAR-X  Changes  Proposed  by  Lawmakers,”  Inside  the  Air  Force,  October 
28,  2005. 

8  Martin  Matishak,  “Lawmakers  Want  Detailed  Cost,  Schedule  Plans:  CSAR-X  Request  Cut  by  $42  Million,  Stipulations 
Faced  on  FY06  Funding,”  Inside  the  Air  Force,  December  23,  2006. 

9  “CSAR-X  Request  Cut  by  $42  Million,  Stipulations  Placed  on  FY06  Funds,”  Inside  the  Air  Force,  December  23,  2005. 

10  The  FAR  defines  “Best  Value”  acquisition  as  one  where  “the  expected  outcome  of  an  acquisition,  in  the  Government’s 
estimation,  provides  the  greatest  overall  benefit  in  response  to  the  requirement”  (U.S.  General  Services  Administration, 
Department  of  Defense,  and  National  Aeronautics  and  Space  Administration,  2005).  (Includes  Amendments  from  Fed¬ 
eral  Acquisition  Circular  [FAC]  2005-42  Effective  June  16,  2010,  FAC  2005-22  December  24,  2007.)  2.1-2.  A  Best  Value 
Source  Selection  can  be  more  specifically  defined  as  “a  process  used  in  Competitively  Negotiated  contracts  to  select  the  most 
advantageous  offer  to  the  government  by  evaluating  proposals  based  on  specifically  identified  non-pricing  criteria  as  well  as 
cost  or  price!  Loretta  Shanks,  “Best  Value  Source  Selection,”  briefing,  June  30,  2009. 
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and  cost/price),  the  Air  Force  identified  the  most  important  discriminators  as  residing  among 
the  mission  capability/proposal  risk  subfactors,  and  past  performance.  The  Boeing  proposal 
scored  significantly  higher  than  the  other  two  competitors  under  the  Block  0  performance 
subcategory,* 11  although  it  was  ranked  as  having  the  same  risk  as  the  Sikorsky  submission.12  The 
Air  Force  also  ranked  Boeing’s  submission  higher  than  the  Lockheed  proposal  under  past  per¬ 
formance  but  equal  to  the  Sikorsky  offering.  On  the  matter  of  cost/price,  GAO  reported  that 
the  Air  Force  SSA  determined  that  the  cost/price  category  was  not  a  significant  discriminating 
factor,  because  the  total  estimated  most  probable  life  cycle  costs  (MPLCC,  which  included 
estimated  O&S  costs)  for  all  three  offerors  were  so  similar.13 

At  the  time,  the  defense  press  widely  speculated  that  Boeing  won  at  least  in  part  because 
its  existing  MH-47G  Chinook  baseline  for  the  HH-47  proposal  would  require  the  least  amount 
of  modification  and  development  cost  and  risk  to  meet  the  CSAR-X  requirements,  especially 
for  the  initial  Block  0  variant.  In  other  words,  it  was  widely  thought  that  the  Boeing  submis¬ 
sion  best  responded  to  the  congressional  budget  and  cost  restrictions  and  other  mandated 
requirements  for  an  off-the-shelf  solution.14 

The  Air  Force  debriefed  Lockheed  and  Sikorsky  on  the  source  selection  process  and  out¬ 
come  on  November  15.  Only  two  days  later,  Sikorsky  submitted  a  formal  bid  protest,  followed 
by  numerous  supplementary  grounds  for  protest.  On  November  20,  Lockheed  joined  Sikorsky 
with  an  initial  bid  protest,  also  followed  by  multiple  supplemental  grounds  for  protest.  GAO 
held  the  CSAR-X  bid  protest  hearings  in  late  January  and  followed  with  post-hearing  com¬ 
ments  and  rebuttals.  Depending  on  how  they  are  tabulated,  more  than  100  grounds  for  protest 
were  filed  by  the  losing  contractors  during  a  period  of  several  weeks  following  the  initial  pro¬ 
test.  Of  all  these  grounds  for  protest,  GAO  sustained  only  one,  but  that  was  sufficient  to  halt 
the  program  dead  in  its  tracks.  GAO  sustained  the  protest  on  February  26,  2007. 15 


11  The  CSAR-X  program  had  adopted  a  two-phase  evolutionary  acquisition  strategy  for  procurement  of  two  basic  blocks 
or  versions  of  the  aircraft  to  speed  delivery  of  the  initial  aircraft  to  the  user.  The  basic  Block  0  version  would  have  needed 
threshold  capabilities,  whereas  the  more  advanced  Block  10  version,  which  would  be  procured  later  in  the  program,  would 
have  considerably  improved  avionics  and  other  equipment  and  capabilities. 

12  The  Air  Force  assigned  each  mission  capability  subfactor  rating  its  own  risk  factor  rating. 

13  U.S.  Government  Accountability  Office,  2007d. 

14  Graham  Warwick,  “Why  Boeing’s  HH-47  Chinook  Won  the  CSAR-X  Competition,”  Flight ,  Flightglobal.com,  October 

11,  2006.  However,  under  the  official  evaluation  of  the  management/schedule  subcategory,  the  Boeing  proposal  received  the 
same  overall  risk  evaluation  as  the  other  two  contractors. 

13  According  to  several  senior  Air  Force  officials  we  interviewed,  the  two  losers — Sikorsky  and  Lockheed  Martin — had  a 
great  deal  to  gain  by  simply  disrupting  and  delaying  the  procurement.  By  default,  Sikorsky  would  become  the  winner  if  the 
Air  Force  could  not  buy  the  HH-47.  It  would  be  able  to  sell  more  UH-60  (Black  Hawk)  variants,  at  least  in  the  interim. 
At  the  same  time,  these  Air  Force  observers  alleged  that  any  additional  time  played  in  Lockheed  Martin’s  favor.  It  would 
provide  Lockheed  Martin  with  an  opportunity  to  submit  a  revised  proposal  to  update  its  technical  offering.  None  of  this 
had  anything  to  do  with  the  life  cycle  cost  (LCC)  estimate.  Thus,  although  the  protest  decision  suggests  that  the  problem 
was  with  the  LCCs,  some  Air  Force  observers  claimed  that  the  real  objective  for  the  two  protesting  vendors  was  to  delay 
the  procurement  if  they  could  not  win.  These  collateral  motives  of  the  two  protesters  are  not  discernible  from  the  GAO 
decision.  These  assertions  do  not  seem  implausible  to  us,  but  we  found  no  decisive  evidence  to  conclusively  support  this 
interpretation. 
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Sustain  #  1:  USAF  Failed  to  Follow  RFP 
Criteria  When  Evaluating  Costs 

•  RFP  required  detailed  data  on  operating  and  support  (O&S) 
costs,  but  noted  USAF  would  calculate  costs  of  Unit  Mission 
Personnel,  Training  Munitions,  and  Indirect  Support 

-  Unit  Mission  Personnel  and  Indirect  Support  accounted 
for  most  of  O&S  cost  totals 

-  USAF  used  same  totals  for  all  three  offerors  based  on 
Manpower  Estimate  Report  (MER) 

•  L-M  and  Sikorsky  protest:  This  violates  RFP  criteria. 

•  GAO:  Request  for  detailed  O&S  cost  data  implied  that  USAF 
would  calculate  unique  O&S  costs  for  each  proposal. 

■  USAF:  (1)  USAF  has  always  used  MER  data  to  evaluate  O&S 
costs.  (2)  USAF  presented  approach  to  offerors  in  briefings 
during  source  selection.  (3)  Issue  is  immaterial. 

•GAO:  (1)  USAF  evaluation  did  not  follow  RFP.  (2)  USAF 
should  reopen  discussions  and  reevaluate  FPRs. 

\|D 
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GAO  sustained  one  ground  for  protest,  saying  that  the  Air  Force  failed  to  follow  the 
stated  RFP  evaluation  criteria  for  calculating  O&S  costs  in  support  of  its  estimate  of  MPLCC. 
Although  this  is  clearly  a  key  issue,  it  is  important  to  note  that  a  minute  detail  in  the  RFP  led 
GAO  to  sustain  the  protest.  In  essence,  one  sentence  in  an  attachment  to  Section  L  of  the  RFP 
permitted  GAO  to  redefine  how  the  Air  Force  should  have  calculated  support  costs.  GAO 
argued  that  the  RFP  request  for  extensive  maintenance  documentation  from  each  contractor 
implied  that  each  competing  aircraft  platform  proposal  would  be  evaluated  separately  with 
respect  to  its  own  unique  O&S  costs  in  support  of  calculating  MPLCC,  which  was  one  of 
four  main  evaluation  factors.  GAO  concluded  that  the  Air  Force  failed  to  follow  this  criterion. 
GAO’s  reasoning  and  the  Air  Force  responses  follow. 

GAO  noted  that  Section  M  of  the  RFP  defined  estimated  O&S  costs  as  part  of  the  cal¬ 
culation  of  MPLCC  for  each  of  the  competing  systems.16  However,  Section  M  did  not  define 
how  O&S  costs  were  to  be  calculated.  In  addition,  Section  L  of  the  RFP  included  an  Attach¬ 
ment  13,  which  was  an  O&S  Data  Form.  The  “primary  purpose”  of  this  data  form  was  “to 
capture  all  relevant  CSAR-X  O&S  costs.”17  Attachment  13  noted,  however,  that  the  Air  Force 
would  itself  calculate  unit  mission  personnel,  training  munitions,  and  indirect  support  costs. 
However,  neither  Attachment  13  nor  any  other  part  of  the  RFP  stated  how  the  government 
planned  to  do  this. 


16  U.S.  Air  Force,  Request  for  Proposal for  the  Combat  Search  and  Rescue  Recovery  Vehicle  (CSAR-X)  System  Development  and 
Demonstration  (SDD),  CSAR-X  Low  Rate  Initial  Production  (LRIP)  and  Initial  Production  Options,  Solicitation  FA8629- 
06-R-2350,  Wright-Patterson  AFB,  Ohio:  Aeronautical  Systems  Center,  2005,  §M.13.2. 

17  U.S.  Air  Force,  2005,  §  L  Att.  13. 
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GAO  concluded  that,  from  these  statements  in  Sections  M  and  L  of  the  RFP,  it  was 
reasonable  for  the  offerors  to  assume  that  the  unique  costs  for  each  offeror’s  aircraft  proposal 
would  be  calculated  separately  and  reflect  the  unique  attributes  of  each  offeror’s  design. 

The  bulk  of  the  O&S  cost  totals  for  each  aircraft  was  driven  by  the  estimated  unit  mission 
personnel  and  indirect  support  costs.  The  Air  Force  ended  up  calculating  the  cost  of  unit  mis¬ 
sion  personnel,  or  the  primary  base-level  personnel  and  other  support  personnel  required  for 
each  squadron,  by  using  actual  data  from  MERs  from  March  2006,  which  obviously  reflected 
current  maintenance  concepts  and  manpower  requirements  for  the  existing  HH-60  helicopter. 
As  a  result,  the  Air  Force  essentially  calculated  the  same  O&S  costs  for  all  three  contractor 
proposals.18 

In  short,  the  methodology  employed  by  the  Air  Force  to  evaluate  the  O&S  costs  as  a  com¬ 
ponent  of  the  MPLCC  did  not  take  into  account  the  unique  cost-related  capabilities  of  each 
offeror’s  design.  Therefore,  GAO  concluded  that  the  Air  Force  failed  to  follow  the  evaluation 
criteria  in  the  FRP  and  thus  sustained  the  protest. 

During  the  process,  the  Air  Force  employed  numerous  arguments  in  an  attempt  to  coun¬ 
ter  GAO’s  position.  For  example,  the  Air  Force  argued  that  the  detailed  O&S  cost  data  and 
reliability  and  maintenance  metrics  requested  by  the  RFP  were  not  tied  directly  to  calculation 
of  the  MPLCC  but  rather  were  associated  with  documenting  whether  the  contractors’  propos¬ 
als  met  the  performance  and  reliability  thresholds  and  objectives  laid  out  in  the  SRD.  The 
Air  Force  also  noted  that  the  MER  is  the  official  baseline  planning  document  for  determin¬ 
ing  staffing  levels  and  has  been  routinely  used  the  same  way  in  many  other  source  selections. 
The  Air  Force  further  argued  that  at  least  one  contractor  was  explicitly  informed  during  an 
interim  briefing  exactly  how  the  Air  Force  planned  to  use  the  MER  to  calculate  O&S  costs. 
Finally,  the  Air  Force  pointed  out  that,  even  if  all  the  contractor  arguments  were  accepted  and 
the  Air  Force  recalculated  unique  O&S  costs  for  each  offeror  based  entirely  on  contractor  data 
(which  the  Air  Force  considered  less  than  fully  reliable),  the  result  would  not  have  dramatically 
changed  the  bottom-line  overall  cost/price  evaluation  for  each  contractor  and,  thus,  would 
have  had  no  material  effect  on  the  outcome  of  the  competition. 

In  short,  at  the  end  of  the  day,  the  protest  turned  on  specific  technical  language  in  an 
obscure  part  of  the  proposal  that,  in  the  view  of  GAO,  implied  an  evaluation  approach  to  one 
small  part  of  the  MPLCC  estimate  that  the  Air  Force  believed  had  no  material  effect  on  the 
outcome  of  the  competition. 

As  a  result  of  the  sustainment  of  the  protest,  GAO  recommended  that  “the  Air  Force 
amend  the  solicitation  to  clarify  its  intent  with  respect  to  the  evaluation  of  O&S  costs,  reopen 
discussions  with  offerors  consistent  with  our  conclusions  above,  and  then  request  revised 
proposals.”19 


18  The  protesters  argued  that  their  aircraft  were  much  newer  designs  and  smaller  than  the  Boeing  MH-47  and  thus  would 
likely  have  lower  operating,  support,  and  maintenance  costs.  These  advantages,  they  argued,  were  left  out  entirely  by  the 
Air  Force  methodology,  thus  unfairly  affecting  the  source  selection  outcome. 

19  U.S.  Government  Accountability  Office,  2007d. 
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Sustain  #2:  USAF  Improperly  Prohibited 
Full  Revision  of  Proposals 

•  USAF  amended  RFP  to  clarify: 

-  USAF  would  base  evaluation  of  Unit  Mission  Personnel 
and  Indirect  Support  cost  on  MER  data 

-  USAF  would  use  proposal-specific  data  to  assess 
supportability  elsewhere  in  evaluation 

•  Based  on  amended  RFP,  USAF  moved  to  re-evaluate 
previously  submitted  proposals  without  revision 

•  L-M,  then  Sikorsky  protested:  If  they  had  understood  this 
approach,  they  would  have  offered  different  proposals 

•  GAO:  Removal  of  proposal-specific  O&S  costs  from  cost 
evaluation  was  a  material  change 

-  GAO  precedent  required  that  offerors  should  be  allowed 
to  revise  their  proposals. 

-  GAO’s  suggested  corrective  action  for  first  sustained 
protest  still  stood. 

MD 
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For  a  variety  of  reasons,  the  CSAR-X  system  program  office  (SPO)  and  the  Air  Force 
acquisition  leadership  came  under  intense  political  and  bureaucratic  pressure  following  the 
GAO  decision  to  sustain  the  protest.  There  were  pressures  within  the  Air  Force  from  the  user 
communities  and  elsewhere  to  minimize  the  delays  to  the  program  and  get  it  moving  quickly 
again.  In  addition,  the  long-anticipated  KC-X  competition  began  heating  up  in  January  2007 
when  the  Air  Force  issued  its  RFP  for  a  new-generation  aerial  refueling  tanker.  The  Air  Force 
hoped  to  quickly  and  decisively  resolve  the  CSAR-X  protests  before  the  battle  lines  were  fully 
drawn  up  between  Boeing  and  Northrop  Grumman/EADS  over  the  tanker  competition,  a 
competition  that  was  expected  to  be  highly  contentious. 

Both  the  CSAR-X  and  the  KC-X  competitions  were  becoming  politicized  as  the  country 
entered  into  the  midst  of  the  presidential  primary  season.  Key  senators  involved  in  the  presiden¬ 
tial  primaries  had  a  direct  interest  in  each  of  the  three  competitors  for  CSAR-X.20  Senator  Hill¬ 
ary  Rodham  Clinton  held  high  visibility  hearings  on  the  program  in  early  March,  vigorously 
questioning  Secretary  of  the  Air  Force  Michael  Wynne  and  Air  Force  Chief  of  Staff  General 
Michael  Moseley  about  the  Air  Force  approach  to  the  CSAR-X  source  selection.  The  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD[AT&L])  held  a  major 
press  conference  emphasizing  the  need  to  rapidly  resolve  the  protests  and  get  the  program  back 
on  track.  As  one  respected  political  journal  rightly  summed  up  at  the  time,  the  GAO  ruling 
sustaining  the  protests  came  “after  months  of  heated  controversy  and  increased  congressional 


20  Senator  Hillary  Clinton  (D-NY),  who  was  a  leading  candidate  for  the  Democratic  presidential  nominee,  and  power¬ 
ful  Senator  Charles  Schumer  (D-NY)  both  took  a  direct  interest  in  the  competition,  since  the  Lockheed  US  101  would  be 
assembled  in  New  York  state.  Two  other  powerful  senators,  Joe  Lieberman  (D/I-CT)  and  Chris  Dodd  (D-CT),  were  strong 
supporters  of  the  Sikorsky  S-92,  which  was  built  in  Connecticut.  Leading  Republican  Senator  Arlen  Spector  supported  the 
Boeing  design,  which  would  be  built  in  his  home  state  of  Pennsylvania. 


58  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and  Options 


scrutiny  over  the  Air  Force’s  decision  to  choose  Boeing’s  CH-47  Chinook  as  the  new  combat 
search-and-rescue  helicopter.”21  The  sustained  protest  made  the  political  background  environ¬ 
ment  only  more  challenging.  Although  CSAR-X  SPO  personnel  told  us  that  they  were  never 
subjected  to  any  kind  of  inappropriate  political  pressure  of  any  type  from  anybody  either  inside 
or  outside  the  Air  Force,  they  nonetheless  had  to  conduct  their  work  against  the  backdrop  of 
a  highly  charged  political  arena. 

Not  surprisingly,  the  Air  Force  adopted  a  position  of  trying  to  resolve  the  protests  as  expe¬ 
ditiously  as  possible  and  avoid  any  precedents  that  could  come  back  to  haunt  the  Air  Force  on 
other  programs  such  as  KC-X.  One  of  the  earliest  actions  the  SPO  took  was  to  seek  reconsid¬ 
eration  from  GAO  regarding  the  multiple  additional  issues  raised  by  the  two  protesters  and  on 
which  GAO  had  not  directly  ruled.  The  Air  Force  requested  reconsideration  in  early  March.  By 
the  end  of  the  month,  GAO  had  replied  that  it  found  no  additional  basis  for  sustained  protests 
based  on  reviewing  the  other  issues  raised  by  both  Lockheed  and  Sikorsky.22 

With  this  finding  in  hand  from  GAO,  the  Air  Force  decided  to  revise  the  RFP  and  limit 
changes  to  the  single  issue  sustained  by  GAO  in  the  first  protest.  The  goal  was  twofold:  to 
clarify  the  original  Air  Force  evaluation  approach  to  O&S  costs  within  the  MPLCC  estimate 
evaluation  and  provide  a  non-quantitative  means  for  the  offerors  to  incorporate  claims  for 
potential  maintenance  manpower  efficiencies  into  the  fourth  evaluation  factor  on  cost/price. 

The  Air  Force  released  RFP  Amendment  4  implementing  these  objectives  on  May  29, 
2007,  and  called  for  the  submission  of  new  proposals  by  June  19.  Amendment  4  made  only 
minimal  changes  to  the  RFP  in  response  to  the  strategy  laid  out  in  the  paragraph  above.  It 
stated  that  O&S  costs  would  still  be  part  of  the  MPLCC  evaluation  for  each  aircraft.  However, 
Amendment  4  explicitly  stated  that  unit  mission  personnel  and  indirect  support  costs  would 
be  calculated  by  the  Air  Force  and  would  be  based  on  the  March  2006  MER  for  all  offerors. 
In  other  words,  O&S  costs  would  be  determined  by  the  current  manning  requirements  of  the 
HH-60  and  would  be  essentially  identical  for  all  three  of  the  competing  offerors’  design  pro¬ 
posals.  The  Air  Force  reasoned  that  GAO  had  never  directly  rejected  this  approach  but  rather 
had  only  ruled  that  it  had  not  been  consistent  with  the  implied  evaluation  criteria  found  in 
other  parts  of  the  original  RFP.  The  Air  Force  therefore  made  its  original  assumptions  and 
approach  explicit. 

However,  the  original  GAO  sustainment  had  been  critical  of  the  failure  of  the  Air  Force 
to  take  into  account  the  potential  support  efficiencies  of  the  newer  and  smaller  designs  offered 
by  the  protesting  competitors.  Therefore,  Amendment  4  permitted  the  offerors  to  document 
the  unique  potential  efficiencies  in  maintenance  tasks  and  reliability  of  their  proposed  designs. 
However,  this  documentation  would  be  evaluated  separately  from  the  MPLCC  evaluation  and 
presented  separately  to  the  SSA. 

Most  important,  the  amended  RFP  did  not  permit  any  other  changes  to  the  original  pro¬ 
posals,  regarding  either  other  cost/price  factors  or  any  other  areas  of  the  offerors  submissions. 

A  little  more  than  a  week  after  the  Air  Force  released  the  revised  FRP  with  Amendment 
4,  Lockheed-Martin  hied  a  new  protest,  followed  shortly  thereafter  by  additional  grounds  for 
protest  as  well  as  new  grounds  for  protest  from  Sikorsky.  Once  again  the  SPO  was  inundated 
with  grounds  for  protest  and  documentation  requirements. 


21  Roxana  Tiron,  “GAO  Rules  in  Favor  of  CSAR-X  Protesters,”  The  Hill ,  August  30,  2007. 

22  U.S.  Government  Accountability  Office,  2007e. 
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On  August  30,  2007,  GAO  announced  that  it  was  once  again  sustaining  one  of  the 
offerors’  grounds  for  protest.23  GAO  ruled  that  the  Air  Force  had  materially  modified  the 
criteria  for  evaluating  O&S  costs  with  Amendment  4  but  had  unreasonably  prohibited  all 
other  changes  to  the  offerors  submissions.  GAO  reasoned  that  the  explicit  removal  by  the  Air 
Force  of  unique  cost  efficiencies  from  the  MPLCC  evaluation  criteria  (by  explicitly  stating  its 
original  approach)  was  a  material  change  to  the  RFP.  Given  this  material  change,  the  offerors 
could  reasonably  expect  to  be  able  to  revise  other  aspects  of  their  proposals  to  compensate  for 
this  change.  For  example,  according  to  GAO,  the  offerors  might  have  wanted  to  make  changes 
in  such  areas  as  technical  approach,  schedule,  and  pricing  to  compensate  for  their  inability  to 
include  potential  cost  efficiencies  in  the  O&S  cost/price  evaluation  subfactor  under  Amend¬ 
ment  4  rules.  By  materially  altering  the  RFP  criteria,  GAO  concluded,  the  Air  Force  was 
required  to  permit  modification  of  any  and  all  aspects  of  the  offerors’  proposals.  Since  the  Air 
Force  explicitly  prohibited  this,  GAO  sustained  the  protest. 

It  is  interesting  to  note  that  GAO  did  not  sustain  the  offerors’  protest  against  the  O&S 
cost  evaluation  ground  rules  and  assumptions  themselves.  The  offerors  once  again  argued  that 
the  Air  Force  evaluation  criteria  minimized  the  potential  O&S  efficiencies  of  the  offerors’ 
newer  and  smaller  aircraft  proposals  by  requiring  the  use  of  the  MER  based  on  the  older 
HH-60  data.  The  Air  Force  argued  that  it  had  to  focus  on  maximum  support  staffing  that 
might  be  required  during  a  crisis  or  wartime  situation  to  determine  cost  risk.  The  offerors 
focused  on  the  lower  total  maintenance  man-hours  normally  required  using  standard  proce¬ 
dures.  GAO  rejected  this  protest,  in  spite  of  its  criticism  of  the  Air  Force  approach  in  the  initial 
appeal  and  sustainment.  GAO  concluded  that  the  Air  Force  emphasis  on  maximum  wartime 
manpower  cost  risk  was  not  an  unreasonable  approach  and  thus  rejected  the  offerors’  protest 
in  this  area. 

In  the  wake  of  the  second  sustained  protest,  the  Air  Force  conducted  numerous  reviews 
of  the  program  and  the  RFR  issuing  three  additional  RFP  amendments,  and  prepared  to 
issue  a  new  RFP  in  the  second  half  of  2009.  Flowever,  on  April  6,  2009,  Secretary  of  Defense 
Robert  Gates  held  a  press  conference  and  directed  the  Air  Force  to  terminate  the  CSAR-X 
program  and  reevaluate  the  requirement  within  a  joint  force  context.  Secretary  Gates  observed 
that  the  CSAR-X  program  had  a  “troubled  acquisition  history”  and  noted  that  DoD  needed 
to  review  the  requirement  to  determine  whether  the  mission  required  a  “specialized”  air¬ 
craft  or  whether  it  should  be  “a  joint  capability”  (that  is,  implemented  by  all  the  services 
together).24  According  to  one  Air  Force-oriented  journal,  such  a  study  by  DoD  “not  only 
threatens  the  Air  Force’s  long-standing  role  as  the  CSAR  specialists,  but  also  complicates  the 
already  challenging  task  of  maintaining  the  dedicated  Air  Force  rescue  forces.  .  .  .  Air  Force 
leaders  had  been  expected  to  announce  the  winner  of  the  second  CSAR-X  competition  in  the 
summer  of  2009.  Now  they  must  wait  to  see  what  the  study  ordered  by  Gates  determines.  .  .  .” 
In  short,  the  Air  Force’s  inability  to  successfully  counter  the  two  CSAR-X  protests  had  con¬ 
tributed  directly  to  the  cancellation  of  the  program  and  led  to  the  possibility  that  the  Air  Force 
would  lose  not  only  a  new  platform  but  also  an  entire  mission  area.25 


23  U.S.  Government  Accountability  Office,  2007c. 

24  Mark  V.  Schanz,  “Gates  Hits  Reset  Button  on  CSAR-X,”  Air  Force  Magazine ,  April  7,  2009. 

25  Otto  Kreisher,  “Is  CSAR  Really  Nothing  ‘Special?’”  Air  Force  Magazine,  November  2009. 
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CSAR-X  Protesters  Filed  100+  Grounds  for 
Protest  Following  Source  Selection  Decision 

•  GAO  denied  all  but  one,  usually  without 
explanation. 

•“Shotgun”  approach  nearly  overwhelmed  relatively 
small,  inexperienced  SPO  staff 

•  Grounds  for  protest  came  in  coordinated  waves, 
each  based  on  responses  to  the  last 

•  SPO  had  little  insight  into  which  of  these  to  take 
seriously 

•  Same  grounds  for  protest  appeared  in  second 
round,  despite  earlier  GAO  denial 

RAND 
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The  CSAR-X  experience  demonstrates  the  potentially  serious  negative  consequences  for 
the  Air  Force  of  sustained  protests  on  Major  Defense  Acquisition  Programs.  These  can  range 
from  public  embarrassment  and  serious  delays  in  delivering  needed  capabilities  to  the  warf¬ 
ighter,  to  outright  cancellation  of  programs  and  even  the  loss  of  control  over  mission  areas. 
Therefore,  it  is  extremely  important  for  the  Air  Force  to  fully  understand  the  environment 
surrounding  protests  and  the  lessons  learned  for  the  future.  What  follows  are  some  specific 
observations  relevant  to  the  CSAR-X  case. 

First,  the  SPO  was  deluged  with  well  over  100  specific  grounds  for  protest.  The  vast 
majority  of  grounds  for  protest  in  this  source  selection  were  given  no  visibility  outside  the 
source  selection  and  have  never  been  fully  documented.  The  immediate  participants  cannot 
even  agree  on  how  many  there  were.  Protesters  were  never  required  to  enumerate  them  in  a 
formal  way. 

There  is  agreement  on  the  following:  The  grounds  for  protest  came  in  waves.  The  Air 
Force  had  to  respond  to  each  wave,  often  by  providing  GAO  and  the  protesters  with  massive 
documentation  to  justify  its  responses.  More  documentation  always  led  to  more  grounds  for 
protest. 

These  protest  waves  appear  to  have  been  preplanned.  So  many  grounds  were  clearly  inap¬ 
propriate  that  it  is  strongly  tempting  to  infer  that  the  protesters  used  a  shotgun  strategy  with 
two  goals  in  mind:  (1)  Try  lots  of  things  and  hope  that  one  would  stick,  and  (2)  overwhelm  the 
program  office,  making  it  more  likely  that  the  Air  Force  would  falter  and  at  least  one  would 
bring  success  from  the  protesters’  perspective. 

The  government  team  was  relatively  small  and  inexperienced  and  had  to  respond  extremely 
rapidly  to  a  very  large  number  of  highly  complex  issues  and  arguments.  By  January  2007,  the 
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protesters  almost  overwhelmed  the  CSAR-X  SPO  just  by  sheer  volume.  The  offerors’  third- 
party  litigation  teams  had  done  source  selection  protests  numerous  times,  and  SPO  staff  was 
doing  it  for  the  first  time.26 

Of  the  grounds  for  protest  received,  there  was  little  strong  agreement  within  the  program 
office  on  which  were  most  likely  to  gain  traction  with  GAO  and  hence  which  deserved  the 
most  attention.  Everyone  was  surprised  by  the  specific  ground  for  protest  upheld  in  the  first 
round,  especially  relative  to  others  that  appeared  more  likely  to  be  selected  by  GAO. 

The  protesters  used  a  similar  approach  in  the  second  protest,  actually  offering  many  of  the 
same  grounds  for  protest  that  GAO  had  denied  in  the  first  round.  In  our  view,  such  behavior 
appears  to  come  very  close  to  fitting  GAO’s  definition  of  “frivolous”  protests  that  might  be 
worth  actively  discouraging. 


26  As  with  much  of  the  material  in  this  chapter,  the  material  in  this  paragraph  is  based  on  interviews  with  individuals  close 
enough  to  the  source  selection  to  base  these  observations  on  their  own  personal  knowledge. 
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Other  Observations  on  CSAR-X 

•  Initial  sustainment  resulted  from  a  simple 
inconsistency  in  RFP  language 

-  Any  ambiguity  opens  the  door  for  GAO  to  ask  what 
a  “reasonable”  approach  would  be. 

-  But  when  explained  in  amended  RFP,  GAO 
accepted  an  approach  it  had  questioned  earlier. 

-Close  technical  scrub  of  RFP  could  have  caught 
this;  vetting  by  high-level,  experienced  experts 
probably  not. 

•Second  GAO  sustain  occurred  despite  thorough, 
high-level  USAF  vetting  of  amended  RFP 

s|D 
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A  second  major  observation  is  that  the  protest  that  GAO  chose  to  uphold  in  the  first 
round  arose  from  ambiguity  about  how  the  Air  Force  would  estimate  costs,  which  in  turn 
gave  GAO  an  opening  to  interpret  what  a  reasonable  approach  to  cost  estimation  should  be. 
Only  an  extremely  rigorous  and  thorough  line-by-line  review  of  the  entire  RFP  and  associated 
documentation  could  have  detected  this  ambiguity.  But  it  is  precisely  just  such  an  in-depth 
and  thorough  review  that  the  protesters  appear  to  have  used  to  identify  candidate  protests.  We 
might  posit  that  they  sought  the  most  vulnerable  spot  in  the  Air  Force  defense  and  targeted 
that  for  effect.27 

By  itself,  the  first  protest  did  not  clarify  how  much  corrective  action  the  Air  Force  would 
have  to  take  to  move  forward.  GAO  ultimately  responded  to  an  Air  Force  request  for  clarifica¬ 
tion  through  reconsideration  by  ruling  that  GAO  had  found  no  other  valid  grounds  for  protest 
in  the  Air  Force  execution  of  its  source  selection.  In  effect,  it  approved  all  other  elements  of  the 
Air  Force  approach. 

Once  GAO  confirmed  that  no  other  issue  raised  by  the  protesters  had  merit,  the  Air  Force 
believed  it  was  free  to  determine  the  appropriate  way  forward  by  addressing  only  the  sustained 
issue.28  The  Air  Force  considered  a  full  range  of  options  for  approaching  the  new  competition 
following  the  original  sustained  protests  and  eventually  elected  to  use  a  higher-risk  strategy  for 
the  follow-on  competition.  The  reasons  for  this  decision  are  a  bit  complex. 


2  In  an  interaction  with  two  players,  such  as  the  Air  Force  and  the  protesters,  such  a  strategy  allows  the  protesters  to  maxi¬ 
mize  the  minimum  payout  they  earn  from  the  game.  In  our  setting,  such  a  strategy  leads  them  to  look  for  the  places  where 
the  Air  Force  is  weakest  and  target  those  places  for  special  attention.  The  protesters  have  to  prevail  on  only  a  single  point  to 
have  GAO  sustain  a  protest. 

28  This  discussion  draws  on  interviews  with  several  Air  Force  officials  with  direct  knowledge  of  the  events  described  here. 
They  also  provided  some  primary  documents  that  support  the  account  given  here. 


Protests  Sustained  in  the  CSAR-X  and  KC-X  Source  Selections  63 


During  the  course  of  the  protest,  a  significant  amount  of  technical  information  regard¬ 
ing  the  competing  CSAR-X  offers  had  been  released,  through  legitimate  redacted  releases,  the 
debriefing  of  the  unsuccessful  offerors,  and  anonymous  leaks  in  the  trade  press.  The  technical 
information  in  the  public  domain  tended  to  be  from  the  successful  offer.  There  was  significant 
concern  within  the  Air  Force  that  the  unsuccessful  offerors  could  be  given  an  inappropriate 
advantage  as  a  result  of  these  disclosures  if  they  had  an  opportunity  to  use  this  new  informa¬ 
tion  to  revise  their  offers.  Both  unsuccessful  offerors  made  it  extremely  clear,  by  both  manage¬ 
ment  and  counsel  communication,  that  they  were  fully  prepared  to  delay  the  source  selection 
with  further  litigation,  should  the  program  office  choose  to  attempt  to  level  the  playing  field 
(i.e.,  by  providing  the  successful  offeror  with  information  regarding  the  unsuccessful  offers 
equivalent  to  that  they  had  received  on  the  successful  offer).  Additionally,  after  the  reconsid¬ 
eration  decision,  pressure  built  within  the  Air  Force  to  quickly  address  the  GAO  decision  and 
move  this  critical  program  forward. 

The  Air  Force  wanted  to  move  forward  quickly  while  mitigating  the  potentially  negative 
effects  of  the  leaks  on  the  competition.  Among  the  many  options  considered,  a  simple  way 
forward  presented  itself:  (1)  Clarify  that  the  Air  Force  would  estimate  the  costs  of  military 
manpower  included  in  the  MPLCC  factor  based  on  its  own  cost  information,  contained  in  the 
MER.  (2)  Add  a  separate  factor  that  would  reflect  each  offeror’s  information  on  the  manpower 
required  to  support  its  proposed  aircraft,  and  have  the  source  selection  weigh  this  information 
subjectively  and  separately  from  the  information  on  MPLCC.  (3)  Ask  offerors  to  adjust  their 
proposals  to  reflect  these  changes  without  changing  the  characteristics  of  the  aircraft  they 
offered.  This  solution  addressed  GAO’s  immediate  concerns,  limited  the  losers’  ability  to  ben¬ 
efit  from  using  leaked  information  to  improve  their  offers,  and  allowed  a  quick  response  that 
would  speed  the  source  selection  forward. 

It  was  clearly  attractive,  but  Air  Force  attorneys  cautioned  the  leadership  that  it  was 
also  risky.  GAO  could  overrule  the  Air  Force  effort  to  prevent  the  offerors  from  responding 
more  fully.  The  Air  Force’s  review  of  relevant  precedent  told  the  attorneys  that  this  risk  was 
acceptable.  GAO  had  ruled  earlier  that  “details  of  implementing  recommendations  of  our 
Office  are  within  the  sound  discretion  and  judgment  of  the  contracting  agency.  Partnership 
for  Response  and  Recovery,  B-298443.4,  Dec.  18,  2006,  2006  CPD  para.  3  at  3.”  But  GAO 
typically  favored  implementation  that  did  not  constrain  offerors’  ability  to  respond  to  amend¬ 
ments  of  the  RFP.  In  one  earlier  case,  GAO  offered  only  two  circumstances  in  which  it  would 
not  require  an  agency  to  allow  this:  “The  agency  offers  evidence  [1]  that  the  amendment  could 
not  reasonably  have  any  effect  on  other  aspects  of  proposals,  or  [2]  that  allowing  such  revisions 
would  have  a  detrimental  impact  on  the  competitive  process.  Cooperativa  Muratori  Riuniti, 
B-294980.5,  July  27,  2005,  2005  CPD  para.  144  at  7.”  The  Air  Force  built  a  case  for  constrain¬ 
ing  responses  based  on  both  grounds  for  exemption. 

On  the  first  point,  the  Air  Force  argued  that  the  protesters  had  already  optimized  their 
aircraft  to  save  maintenance  manpower.  The  new  source  selection  factor  would  now  give  them 
a  clear  way  to  reflect  that  optimization  in  their  individual  proposals.  Why  should  they  change 
the  designs  they  proposed?  On  the  second  point,  leaks  had  given  the  unsuccessful  offerors 
extensive  technical  knowledge  about  the  winner.  Allowing  complete  proposal  revisions  would 
have  a  detrimental  effect  on  the  competitive  process  and  jeopardize  the  integrity  of  the  source 
selection  process.  The  Air  Force  believed  the  limited  reopening  corrective  action  was  firmly 
based  in  prior  GAO  precedent  and  on  careful  examination  of  the  context  of  the  original  deci¬ 
sion  and  a  close  reading  of  GAO  case  law. 
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On  April  20,  2007,  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition  was  briefed  on 
this  potential  corrective  action.  She  had  previously  been  briefed  on  a  wide  range  of  potential 
actions  and  in  light  of  the  reconsideration  decision  had  asked  for  more  details  on  this  action. 
This  briefing  included  an  analysis  of  the  various  risks  involved  in  choosing  this  path  forward. 
Of  particular  note,  all  risks  were  designated  as  “low”  except  for  limiting  re-proposals.  That  risk 
was  briefed  as  “low-moderate,”  but  the  verbal  briefing  went  further.  It  was  specifically  stated 
that  the  protesters  would  focus  almost  all  their  effort  and  time  on  that  particular  issue  and  thus 
the  risk  might  be  higher.  The  briefer  specifically  stated  that  the  re-proposal  issue  would  be  “the 
protester’s  Little  Bighorn  and  Alamo  rolled  into  one.” 

The  decision  to  limit  revisions  of  proposals  was  a  corporate  decision  believed  at  the  time 
by  all  involved  to  be  in  the  best  interests  of  the  Air  Force  and  the  course  of  action  most  likely 
to  preserve  the  integrity  of  the  source  selection  process.  According  to  our  interviewees,  senior 
Air  Force  leaders  had  a  clear  understanding  of  the  risks  associated  with  the  chosen  approach. 
They  understood  at  the  time  that  restrictions  on  the  offeror’s  ability  to  respond  to  the  amended 
RFP  would  be  almost  the  sole  focus  of  any  further  protests  and  they  would  fight  very  hard. 

GAO  disagreed  with  the  Air  Force  assessment  in  two  qualitatively  different  ways.  First, 
the  protesters  both  claimed  that  they  would  have  changed  the  designs  they  had  offered  if  they 
had  been  allowed  to  reflect  the  lower  priority  the  Air  Force  now  appeared  to  give  aircraft  main¬ 
tenance  requirements.  In  effect,  they  argued  that  they  had  optimized  against  one  set  of  crite¬ 
ria  in  the  original  RFP  and  would  have  optimized  differently  if  they  had  known  the  criteria 
offered  in  the  amended  RFP.  The  Air  Force  did  not  refute  these  claims  to  GAO’s  satisfaction. 

Second,  GAO  cited  precedent  that  materiality  by  itself  required  that  the  Air  Force  not 
constrain  how  the  offerors  responded  to  the  amended  RFP: 

It  is  fundamental  that,  where  an  agency  materially  changes  the  solicitation’s  evaluation 
scheme,  offerors  must  be  given  a  reasonable  opportunity  to  respond  to  the  revised  scheme; 
otherwise,  the  statutory  requirement  to  notify  offerors  of  the  criteria  upon  which  their 
offers  will  be  evaluated  is  meaningless.  Dept,  of  Commerce — Request  for  Modification  of 
Recommendation,  B-283137.7,  Feb. 14,  2000,  2000  CPD  para.  27  at  3.  In  these  circum¬ 
stances,  we  conclude  that  the  Air  Force,  having  materially  altered  the  methodology  for  eval¬ 
uating  O&S  costs,  was  therefore  required  to  permit  offerors  to  revise  both  the  cost/price 
and  non-cost/price  aspects  of  their  proposals  in  response  to  the  new  evaluation  scheme. 

This  precedent  from  2000  basically  argued  that,  in  the  presence  of  a  material  effect,  it  is  irrel¬ 
evant  whether  an  offeror’s  full  response  to  the  amended  RFP  would  compromise  the  integrity 
of  the  source  selection.  Only  materiality  matters. 

Thus,  the  highest-risk  approach  was  selected  for  a  variety  of  complex  but  justifiable  rea¬ 
sons  but  proved  to  be  a  mistake  given  GAO’s  ultimate  response. 

Next,  we  turn  to  the  largest  and  best  known  of  the  recent  major  high-profile  protests 
recently  filed  against  the  Air  Force:  the  KC-X  program. 
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Sustained  KC-X  Protests  Added  Delays  to 
Long-Delayed  Tanker  Recapitalization 

•  2003-04:  USAF  attempts  to  lease  new  tankers  from  Boeing 
and  fails. 

•  2004-06:  AoA  clarifies  tanker  options,  delays  competition. 

•  2006:  USAF  formally  initiates  KC-X  program. 

•  Sep  2006:  USAF  issues  draft  RFP. 

•  Feb  2008:  Northrop  Grumman  (N-G)/EADS  wins.  Boeing 
protests. 

•  Jun  2008:  GAO  sustains  multiple  grounds  for  protest  (#1-8), 
denies  the  remainder. 

•  Sep  2008:  SecDef  terminates  competition,  defers  to  next 
administration. 

•  Late  2009:  USAF  issues  amended  draft  final  RFP. 

AND 

GAO  Bid  Protests  in  Air  Force  Source  Selections  -28  - 


Aerial  Refueling  Tanker  Aircraft  Program 

The  KC-X  program  emerged  from  a  long,  convoluted,  and  difficult  series  of  events  stretching 
over  a  decade  or  more.  At  least  as  far  back  as  the  conclusion  of  the  first  Gulf  War  in  early  1991, 
government  airpower  experts  and  aircraft  manufacturers  began  seriously  examining  possible 
options  for  replacing  the  Air  Force’s  aging  fleet  of  KC-135  aerial  refueling  tankers,  because  of 
rising  support  costs  and  declining  reliability.29  As  early  as  July  1992,  Boeing  publicly  discussed 
the  use  of  the  Boeing  767-200ER  as  the  baseline  for  a  new  KC-135  replacement.30  While  rec¬ 
ognizing  the  need  for  replacing  or  upgrading  the  tanker  aircraft,  the  Air  Force  initially  focused 
on  other  higher-priority  procurements  during  a  period  of  very  constrained  budgets.  In  1996, 
this  began  to  change,  however,  when  GAO  published  a  report  critical  of  AMC’s  decision  to 
defer  replacement  of  the  KC-135  until  the  2013  time  frame,  because  of  the  growing  costs  of 
maintaining  and  operating  the  aging  aircraft.31 

Beginning  in  March  2000,  Boeing  established  a  special  business  unit  and  began  lobby¬ 
ing  Congress  and  actively  marketing  the  767  tanker  to  the  U.S.  and  foreign  governments.  This 
marketing  activity  became  particularly  urgent  to  Boeing  in  a  commercial  sense  in  view  of  the 
severe  downturn  in  commercial  aircraft  sales  after  2000  and  the  looming  prospect  of  having 


29  The  Boeing  KC-135,  based  on  the  Boeing  707  airliner  design,  first  entered  Air  Force  service  in  1957. 

30  Congressional  Research  Service,  The  Air  Force  KC-767  Tanker  Lease  Proposal:  Key  Issues  for  Congress,  Washington,  D.C.: 
The  Library  of  Congress,  RL32056,  updated  September  2,  2003. 

31  U.S.  General  Accounting  Office,  U.S.  Combat  Air  Power:  Aging  Refueling  Aircraft  Are  Costly  to  Maintain  and  Operate, 
GAO/NSIAD-96-160,  Washington,  D.C.,  August  1996.  Also  see  Amy  Butler,  “Air  Force  Mulling  Replacement  for  Aging, 
Maintenance-Needy  KC-135,”  Inside  the  Air  Force,  May  4,  2001. 
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to  shut  down  the  767  commercial  airliner  production  line.  Congressional  pressure  increased 
on  the  Air  Force.  For  example,  in  September  2001,  Representative  Norman  Dicks  (D-WA),  a 
member  of  the  Defense  Subcommittee  of  the  House  Appropriations  Committee,  held  a  press 
conference  where  he  announced  that  he  planned  to  “insert  an  amendment  into  a  defense 
appropriations  bill  to  jump-start  the  Air  Force’s  purchase  of  hundreds  of  Boeing  767  tankers.”32 
Dicks  represents  Washington’s  6th  Congressional  District,  which  includes  Boeing’s  main  com¬ 
mercial  aircraft  assembly  plant  at  Everett  where  the  767  was  assembled. 

In  1999,  AMC  formally  began  examining  options  for  replacing  the  KC-135  in  its  Air 
Mobility  Strategic  Plan  2000 ,33  By  October  2000,  AMC  had  begun  to  draft  a  MNS  for  a  future 
aerial  tanker.  In  February  2001,  several  tanker  requirements  studies  were  undertaken.  The 
same  month,  Boeing  made  an  unsolicited  offer  to  sell  thirty  767  tankers  to  the  Air  Force  as  a 
stop-gap  measure  pending  completion  of  a  formal  AoA.34 

Following  the  September  11  terrorist  attacks,  OSD  established  a  special  leasing  panel  to 
review  leasing  options  to  rapidly  procure  needed  capabilities  in  support  of  the  war  on  terrorism 
and  other  new  priorities.  By  the  end  of  the  year,  the  Secretary  of  the  Air  Force  revealed  that  the 
service  was  discussing  a  deal  with  Boeing  to  lease  one  hundred  767  tankers.  The  justification 
for  leasing  was  to  save  money  and  time  to  bring  the  needed  capability  into  the  force  structure 
as  quickly  as  possible  and  with  the  least  up-front  cost.  Things  now  moved  quickly.  By  Janu¬ 
ary  2002,  special  emergency  congressional  legislation  in  response  to  the  September  11  attacks 
authorized  the  one  hundred  767  tanker  lease  program  with  a  contract  award  planned  for  June 
2002.35 

As  detailed  negotiations  progressed  between  the  Air  Force  and  Boeing  over  the  lease  of 
the  one  hundred  767  tankers,  skepticism  and  outright  opposition  to  the  deal  grew  in  Congress 
and  elsewhere,  largely  driven  by  growing  concerns  that  the  lease  would  cost  more  than  an  out¬ 
right  purchase  and  that  the  Air  Force  had  not  fully  evaluated  its  requirements  or  conducted  a 
thorough  analysis  of  alternatives.  Some  critics  believed  that  the  proposed  leasing  deal  violated 
acquisition  regulations,  did  not  contain  adequate  cost  transparency,  and  undermined  congres¬ 
sional  oversight.  Finally,  other  critics  were  opposed  to  the  lease  proposal  because  it  lacked  com¬ 
petitive  bids  and  was  essentially  a  sole-source  contract. 

Throughout  2002  and  2003,  a  complex  and  convoluted  history  of  congressional  hearings; 
various  studies  conducted  by  GAO,  federally  funded  research  and  development  centers,  and 
other  organizations;  ongoing  negotiations  between  the  Air  Force  and  Boeing;  and  continued 
Air  Force  development  of  the  ORD  combined  to  delay  the  award  of  the  leasing  contract.  One 


32  Katherine  Pfleger,  “Lawmakers  Consider  Air  Force  as  Boeing  Commercial-Plane  Customer,”  Associated  Press  Newswires, 
September  30,  2001,  quoted  in  Congressional  Research  Service,  2003. 

33  Amy  Butler,  “Air  Mobility  Command  to  Begin  Search  for  KC-135  Tanker  Replacement,”  Inside  the  Air  Force,  Decem¬ 
ber  17,  1999;  U.S.  Air  Force,  Air  Mobility  Strategic  Plan  2000,  Scott  Air  Force  Base,  Ill.:  Headquarters,  Air  Mobility 
Command. 

34  Boeing  won  a  contract  to  build  four  767  tankers  for  the  Italian  Air  Force  in  April  2000.  A  similar  contract  for  four  tank¬ 
ers  was  later  signed  with  Japan. 

33  U.S.  Department  ofDefense,  Inspector  General,  Acquisition  of  the  Boeing  KC-767A  Tanker  Aircraft,  D-2004-064,  Wash¬ 
ington,  D.C.,  March  29,  2004. 
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of  the  most  important  of  these  studies  was  a  GAO  study,  which  concluded  that  a  lease  agree¬ 
ment  would  cost  more  than  outright  procurement.36 

In  view  of  these  study  findings  and  other  concerns,  Congress  and  outside  observers  con¬ 
tinued  to  sour  on  the  proposed  leasing  agreement.  In  November  2003,  Congress  amended  the 
original  100  aircraft  leasing  deal  in  the  new  National  Defense  Authorization  Act  for  Fiscal 
Year  2004,  calling  for  the  outright  purchase  of  80  of  the  100  KC-767As  using  a  multiyear  pro¬ 
curement  approach.37  At  the  beginning  of  February  2004,  the  Deputy  Secretary  of  Defense 
requested  a  Defense  Science  Board  study  of  the  KC-135  to  reassess  the  requirement.  Three 
weeks  later,  the  acting  USD(AT&L)  directed  the  Air  Force  to  conduct  an  aerial  refueling  AoA. 
DoD  then  placed  the  lease  contract  negotiations  on  hold,  and  the  Deputy  Secretary  of  Defense 
requested  that  the  DoD  Inspector  General  (IG)  review  the  program.  The  DoD  IG  169-page 
report,  released  in  March  2004,  found  that  the  Air  Force  had  “used  an  inappropriate  procure¬ 
ment  strategy  and  demonstrated  neither  best  business  practices  nor  prudent  acquisition  proce¬ 
dures”  in  its  negotiations  of  the  KC-767A  program.38 

The  outcome  of  the  DoD  IG  review,  combined  with  a  burgeoning  new  acquisition  scan¬ 
dal  within  the  Air  Force,  led  to  the  demise  of  the  KC-767A  tanker  lease  program.  The  IG 
review  was  in  part  a  response  to  growing  allegations  of  a  major  Air  Force  procurement  scandal. 
The  allegations  became  fact  in  April  2004,  when  a  former  senior  Air  Force  acquisition  offi¬ 
cial  pleaded  guilty  to  criminal  conspiracy.  The  official,  Darken  Druyun,  had  been  Principal 
Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition  and  Management  and  had  been 
the  lead  negotiator  on  the  767  tanker  lease  deal.  In  November  2002,  she  had  announced  she 
was  leaving  the  Air  Force  for  a  job  as  deputy  general  manager  of  Boeing’s  missile  defense  sys¬ 
tems.  Increasing  allegations  about  alleged  improprieties  led  the  Air  Force  to  place  the  767  lease 
deal  on  hold  in  December  2003. 39  Druyun  later  admitted  in  the  plea  agreement  to,  among 
other  things,  having  agreed  to  a  higher  price  than  was  appropriate  in  the  Boeing  767  lease  deal 
to  cement  her  relationship  with  her  future  employer.40  CBS  news  called  it  “the  biggest  Penta¬ 
gon  scandal  in  20  years.”41 

The  net  effect  of  the  congressional  controversy  and  increased  scrutiny  over  the  lease  pro¬ 
gram,  the  multiple  delays,  the  findings  of  the  IG  investigation,  and  the  revelations  of  the  Air 
Force  acquisition  scandal  were  profound.  Because  of  the  much-increased  critical  scrutiny  by 
Congress  and  GAO  under  which  the  program  now  had  to  operate,  an  open,  fair,  and  trans¬ 
parent  competition  for  outright  procurement,  rather  than  a  sole-source  award  for  a  lease,  now 
became  an  imperative.  To  control  costs,  the  aerial  tanker  would,  of  course,  have  to  be  based  on 
an  existing  commercial  airliner  and  to  ensure  competition,  that  meant  including  the  only  other 
remaining  credible  competitor  in  the  world,  foreign-based  EADS. 


36  U.S.  General  Accounting  Office,  Military  Aircraft:  Observations  on  the  Proposed  Lease  of  Aerial  Refueling  Aircraft  by  the 
Air  Force,  GAO-03-923T,  Washington,  D.C.,  September  4,  2003. 

37  U.S.  General  Accounting  Office,  DoD  Needs  to  Determine  Its  Aerial  Refueling  Aircraft  Requirements,  GAO-04-349, 
Washington,  D.C.,  June  2004. 

38  U.S.  Department  of  Defense,  Inspector  General,  2004,  “Executive  Summary.” 

39  John  A.  Tirpak,  “Tanker  Twilight  Zone,”  Air  Force  Magazine,  February  2004. 

40  George  Cahlink,  “Ex-Pentagon  Procurement  Executive  Gets  Jail  Time,”  Government  Executive,  October  1,  2004. 

41  Rebecca  Leung,  “Cashing  In  for  Profit?  Who  Cost  Taxpayers  Billions  in  Biggest  Pentagon  Scandal  in  Years?”  CBS  News, 
January  5,  2005. 
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Maintaining  the  ongoing  interest  of  offerors  who  would  be  using  preexisting  aircraft 
designs  with  well-known  capabilities  created  deep  tensions  that  persist  to  this  day.  We  com¬ 
ment  further  on  this  below. 

The  competition  for  a  new  tanker  was  further  delayed  by  an  AoA  that  ended  up  shaping 
the  subsequent  competition  in  a  number  of  ways.  For  now,  perhaps  the  most  important  point 
is  that  the  delay  itself  heightened  the  Air  Force’s  desire  to  complete  the  source  selection  as  rap¬ 
idly  as  possible,  creating  schedule  concerns  that  the  Air  Force  has  since  concluded  contributed 
to  the  sustained  protests  to  come.  To  complicate  matters  further,  the  tanker  program  had  now 
slipped  to  a  point  where  it  overlapped  with  another  very-high-profile  competition,  which  has 
already  been  discussed,  the  CSAR-X  program. 

The  KC-X  AoA  was  ultimately  carried  out  by  the  RAND  Corporation  and  completed  in 
March  2006. 42  The  RAND  AoA  concluded  that  a  fleet  of  new  medium  to  large  aerial  tank¬ 
ers  based  on  modified  commercial-derived  passenger  aircraft  would  be  the  most  cost-effective 
solution  to  the  KC-X  requirement.  The  earlier  DoD  IG  program  review  (March  29,  2004)  as 
well  as  congressional  hearings  and  GAO  testimony  had  made  it  clear  that  the  KC-X  program 
had  to  be  a  free  and  open  competition. 

In  April  2006,  the  USD(AT&L)  signed  a  memorandum  authorizing  the  Air  Force  to 
resume  the  formal  KC-X  acquisition  process.  The  KC-X  program  office  issued  the  final  RFP 
to  contractors  on  January  30,  2007,  for  the  procurement  of  179  tankers.43  Boeing  responded 
to  the  RFP  with  the  KC-767,  a  design  based  on  several  different  variants  of  its  commercial  767 
passenger  airliner,  rather  than  a  copy  of  its  existing  KC-767  tanker  for  foreign  customers.44 
EADS,  teamed  with  U.S.  contractor  Northrop  Grumman,  offered  its  KC-30  (later  changed  to 
KC-45)  based  on  a  modification  of  the  Airbus  A330  airliner.45 

The  Air  Force  announced  on  February  29,  2008,  that  the  Northrop  Grumman/EADS 
team  had  won  the  competition.  This  decision  was  viewed  by  the  press  as  a  “stunning  upset,”46 
since  Boeing  “was  widely  expected  to  win  the  contract.”47  Less  than  two  weeks  later,  Boeing 
filed  a  bid  protest  with  GAO.  In  a  decision  viewed  by  many  observers  as  almost  as  surprising  as 
the  original  selection  of  the  Northrop  Grumman/EADS  team,  GAO  announced  on  June  18, 
2008,  that  it  had  sustained  Boeing’s  protest. 

On  July  9,  2008,  the  Secretary  of  Defense  announced  the  cancellation  of  the  KC-X  con¬ 
tract  award  to  Northrop  Grumman/EADS  because  of  the  GAO  decision.  Secretary  Gates 
then  stated  that  a  new  competition  would  take  place,  with  a  new  contract  award  expected  in 
December.  In  a  telling  blow  to  the  Air  Force,  Secretary  Gates  also  announced  that  the  author¬ 
ity  to  conduct  the  new  source  selection  would  be  taken  over  directly  by  the  Department  of 


42  Michael  Kennedy,  “Analysis  of  Alternatives  for  Recapitalizing  the  U.S.  Air  Force  KC-135  Aerial  Refueling  Tanker 
Fleet,”  Santa  Monica,  Calif.:  RAND  Corporation,  unpublished  research,  2006. 

43  This  was  planned  to  be  the  first  of  three  equal  buys  stretching  out  over  many  years  (U.S.  Department  of  Defense,  Inspec¬ 
tor  General,  Air  Force  KC-X  Aerial  Refueling  Tanker  Aircraft  Program,  D-2007-103,  Washington,  D.C.,  May  30,  2007). 

44  Critics  of  the  Boeing  proposal  labeled  the  aircraft  the  “Frankentanker.”  See  Rami  Grunbaum,  “Opponents  Call  Boeing’s 
Plane  ‘Frankentanker,’”  The  Seattle  Times,  January  27,  2008. 

45  For  simplicity,  we  will  refer  to  this  proposed  system  throughout  as  the  KC-45. 

46  Leslie  Wayne,  “U.S. -Europe  Team  Beats  Out  Boeing  on  Big  Contract,”  New  York  Times,  March  1,  2008. 

47  Jeff  Bailey  and  David  M.  Flerszenhorn,  “Boeing  Says  It  Will  Protest  Tanker  Deal,”  New  York  Times,  March  11,  2008. 
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Defense  and  placed  under  the  Under  Secretary  of  Defense  for  Acquisition.48  However,  at  the 
end  of  the  summer,  the  Secretary  of  Defense  decided  to  delay  the  new  competition  until  after 
the  presidential  elections  in  November  to  permit  the  new  president’s  team  to  decide  how  to 
proceed.  The  new  competition  for  the  KC-X  ultimately  did  not  begin  until  September  2009, 
with  a  contract  award  expected  in  the  summer  of  2010.  Thus,  the  sustained  protest  had  delayed 
the  program  at  least  an  additional  year  and  a  half  and  had  led  to  the  third  attempt  in  seven 
years  to  begin  acquisition  by  the  Air  Force  of  a  new  aerial  refueling  tanker. 

The  2008  KC-X  source  selection  protest  resulted  in  eight  sustained  grounds  for  protest 
and  two  additional  decisions  on  procedure  that  went  against  the  Air  Force.  This  large  number 
of  negative  decisions  is  unusual  in  a  single  GAO  protest  decision,  leading  some  to  speculate 
that,  once  GAO  decided  to  sustain  one  ground  for  protest,  it  added  on  others  to  make  its  deci¬ 
sion  unassailable.49  If  this  occurred,  it  is  very  unusual,  at  least  among  the  GAO  decisions  we 
examined  in  detail. 

An  overview  of  the  eight  principle  sustained  grounds  for  protest  is  presented  below. 


48  Marcus  Weisgerber,  “Gates  Reopens  Tanker  Competition,  Shifts  Authority  from  Air  Force,”  Inside  the  Pentagon,  August 
10,  2009. 

49  Several  knowledgeable  people  we  interviewed  offered  this  interpretation. 


70  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and  Options 


Overview  Points  to  Misunderstanding  of  GAO, 
Lack  of  Discipline,  Poor  Costing  Methods 


Sustainment 

Basis  for  GAO 
Decision 

Apparent  Source  of  Failure 

1.  Evaluation  of 
requirements  achieved 

Evaluation 

inconsistent  with  RFP 

Poorly  framed  requirements 
in  RFP 

2.  Evaluation  of 
refueling  capacity 

Evaluation 

inconsistent  with  RFP 

Inconsistent  language  in 

RFP 

3.  Evaluation  of 
compatibility 

Inadequate 

documentation 

Failure  to  close  out  EN 
clearly 

4.  Unequal  treatment 
on  operational  utility 

Unequal  treatment 

Misunderstanding  of  GAO 
rules 

5.  Organic  depot 
support  requirement 

Unreasonable 

evaluation 

Incomplete  preparation  of 
SSAC,  SSA 

6.  Estimation  of 
MILCON  costs 

Unreasonable  cost 
estimation 

Mismatch  between  RFP  and 
evaluation;  inadequate 
costing  methods 

7.  Addition  of  non¬ 
recurrent  costs 

Failure  to  comply  with 
administrative  rules 

Misunderstanding  of  GAO 
rules 

8.  Estimation  of 
nonrecurring  costs 

Unreasonable  cost 
estimation 

Inadequate  costing  methods 
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This  slide  provides  a  very  brief  summary  of  each  of  the  eight  sustained  grounds  for  pro¬ 
test,  including  a  short  description  of  the  basis  of  the  GAO  decision,  and  a  summary  of  our 
assessment  of  the  apparent  source  of  failure  on  the  side  of  the  Air  Force.  Before  we  review  each 
sustained  ground  for  protest  in  greater  detail,  we  provide  several  general  comments  and  over¬ 
view  observations  on  all  eight  of  the  sustainments.  This  is  particularly  important,  because  we 
found  that  a  root  cause  of  many  of  the  problems  that  led  GAO  to  sustain  specific  grounds  for 
protest  was  the  Air  Force’s  failure  to  fully  comprehend  GAO’s  mode  of  operation  and  analysis. 

Our  first  point  is  that  all  the  GAO  decisions  closely  argue  the  basis  for  each  sustained 
ground  for  protest,  offering  careful  arguments  for  each  sustainment  that  are  deeply  informed 
by  GAO’s  own  precedents.  Various  officials  in  the  Air  Force  told  us  that  they  considered 
GAO’s  stated  grounds  for  sustainment,  at  least  on  some  of  the  sustained  grounds  for  protest,  to 
be  highly  technical,  nitpicking,  lacking  common  sense  or  a  sense  of  fair  play,  or  not  properly 
attuned  to  the  priorities  of  the  warfighter.  We  believe  that  such  critiques  miss  a  basic  point.  The 
GAO  decision  applies  a  carefully  documented  body  of  GAO  precedent  to  a  new  situation  with 
precisely  the  intent  of  focusing  on  GAO’s  procedural  priorities  and  avoiding  substantive  argu¬ 
ments  about  what  matters  to  the  warfighter.  As  long  as  GAO  sets  the  effective  rules,  relevant 
Air  Force  decisionmakers  must  clearly  understand  what  the  rules  are. 

Once  a  source  selection  has  such  an  understanding  clearly  in  hand,  to  avoid  sustainments 
such  as  those  experienced  here,  the  Air  Force  must  demonstrate  a  high  degree  of  discipline  in 
the  way  it  crafts  an  RFP,  evaluates  proposals  against  the  RFP,  and  treats  offerors.  Most  protests 
tend  to  arise  from  small  details  that,  if  they  had  been  properly  executed,  would  not  have  led  to 
sustainments. 

Two  of  the  sustainments  (numbers  7  and  8)  arose  from  GAO’s  willingness  to  raise  sub¬ 
stantive  questions  about  the  Air  Force’s  costing  methods.  In  both  cases,  GAO  raised  what 
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appear  to  be  reasonable  concerns  from  a  cost  analyst’s  point  of  view,  but  concerns  that  are  hard 
to  reconcile  with  GAO’s  tendency  to  distance  itself  from  substantive  issues.  And,  in  fact,  GAO 
could  have  raised  comparable  issues  about  the  costing  in  the  CSAR-X  case  and  explicitly  chose 
not  to  once  the  Air  Force  documented  its  costing  approach  clearly  enough  so  that  GAO  had 
no  room  to  speculate  about  what  a  “reasonable”  approach  would  be.50  In  the  KC-X  case,  GAO 
imposed  its  views  on  a  reasonable  approach,  suggesting  that  future  Air  Force  analysts  should 
more  carefully  review  what  GAO  regards  as  reasonable  and  unreasonable. 

As  closely  argued  as  the  GAO  decision  is,  it  tends  not  to  resolve  concerns  we  heard  in 
the  Air  Force  and  elsewhere  about  materiality.  It  is  hard  to  imagine  circumstances  in  which, 
for  example,  issues  raised  in  sustainments  4  or  6  could  actually  have  altered  the  winner  of  the 
competition.  That  said,  a  fair  reading  of  arguments  for  the  other  grounds  for  protest  suggests 
that  any  of  these  might  have  changed  the  outcome.  Issues  raised  in  sustainments  3  and  5  could 
have  had  immediate  effects. 

Below  we  review  each  of  the  eight  KC-X  protests  in  greater  detail. 


5°  Note  the  GAO  decision  on  the  second  CSAR-X  protest  episode  where  GAO  overruled  the  offerors  protest  against  the 
methodology  employed  by  the  Air  Force  using  the  MER  to  determine  aspects  of  the  supports  costs  as  a  component  of  the 


overall  MPLCC. 
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Sustain  #1:  USAF  Did  Not  Evaluate  Number  of 
Requirements  Achieved  as  Indicated  in  RFP 

*  RFP:  Offerors  should  satisfy  as  many  trade-space  technical 
requirements  as  possible.  Evaluators  failed  to  credit  one 
proposal  for  satisfying  more  of  these  than  the  other. 

*  USAF:  Source  selection  based  its  evaluation  on  the  relative 
importance  of  these  requirements  indicated  in  the  RFP. 

*  GAO:  Documentation  from  the  source  selection  did  not 
support  this  USAF  contention.  As  a  result,  GAO  ruled  that 
USAF  evaluation  was  inconsistent  with  criteria  stated  in  RFP. 

*  Takeaway:  Problems  existed  in  RFP  design  and  evaluation 

-  RFP  did  not  restrict  trade  space  to  key  discriminators 

-  RFP  used  imprecise  language  to  state  how  source 
selection  would  evaluate  items  in  trade  space 

-  Documentation  did  not  clearly  link  specific  evaluation 
decisions  to  specific  language  in  RFP. 
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The  first  GAO  sustainment  was  rather  straightforward  and  difficult  to  dispute.  In  short, 
GAO  found  that  the  Air  Force  failed  to  follow  the  clearly  stated  evaluation  criteria  as  laid 
out  in  the  RFP.  This  specifically  applied  to  the  key  system  requirements  subfactor,  the  most 
important  subfactor  of  the  mission  capability  evaluation  factor,  where  Boeing  alleged  it  was 
not  awarded  credit  for  having  met  more  of  the  requirements  in  the  five  assessment  areas  than 
had  Northrop  Grumman/EADS.  A  brief  review  of  the  RFP  evaluation  criteria  is  necessary  to 
understand  GAO  arguments. 

The  RFP  states  that  an  award  would  be  made  on  a  “best  value”  basis  and  proposals  would 
be  judged  with  respect  to  five  main  evaluation  factors.  These  were  mission  capability,  proposal 
risk,  past  performance,  cost/price,  and  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA). 
The  first  three  factors  were  stated  to  be  of  equal  importance,  and  individually  of  more  impor¬ 
tance  than  the  last  two  factors,  which  were  equal  in  weight.  The  mission  capability  factor 
had  five  subfactors:  key  system  requirements,  system  integration  and  software,  product  sup¬ 
port,  program  management,  and  technology  maturity  and  demonstration.  These  subfactors  are 
stated  to  be  of  decreasing  importance,  with  the  key  system  requirements  the  most  important. 
In  addition,  the  proposal  risk  would  be  evaluated  only  as  a  separate  component  of  each  of  the 
first  four  mission  capability  subfactors.51 

The  RFP  also  included  a  detailed  SRD.  The  RFP  notes  that  the  key  system  requirements 
subfactor,  the  most  important  subfactor  under  the  mission  capability  factor,  would  be  assessed 
in  relation  to  the  detailed  technical  requirements  laid  out  in  the  SRD.  These  included  the  abso¬ 
lute  requirement  of  satisfying  key  performance  parameter  (KPP)  “threshold”  capabilities.  A 
proposal  would  be  found  completely  unacceptable  if  it  did  not  meet  all  KPP  threshold  require- 


51  In  other  words,  the  final  evaluated  rating  for  key  system  requirements  would  include  a  second  rating  for  proposal  risk 
and  on  down  the  line  for  the  first  four  mission  capability  subfactors. 
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ments,  of  which  there  were  nine.  The  RFP  also  provided  “objective”  capabilities  for  some  KPPs, 
which  were  portrayed  as  desirable  goals  but  not  absolutely  necessary  and  tradable  against  other 
objectives  and  capabilities. 

Finally,  the  RFP  explained  that  the  SRD  requirements  would  be  evaluated  under  the 
key  system  requirements  subfactor  in  five  key  areas:  aerial  refueling,  airlift,  operational  utility, 
survivability,  and  other  system  requirements.52  The  SRD  also  listed  on  the  order  of  800  other 
requirements  and  attributes,  including  key  system  attributes  (KSA)  that  would  be  evaluated 
under  these  same  five  areas.  Unlike  KPP  thresholds,  these  were  desirable  but  not  required  and 
could  all  be  traded  off  against  each  other  or  against  other  factors  as  part  of  the  offerors’  “design 
space.” 

It  is  necessary  to  grasp  the  basics  of  this  complex  evaluation  structure  to  understand  the 
GAO’s  arguments  justifying  its  first  ground  for  protest  sustainment.  The  statement  of  objec¬ 
tives  of  the  KC-X  SDD  stated  that  the  offerors  must  meet  all  the  KPP  thresholds  and  should 
try  to  meet  as  many  of  the  KSA  and  other  requirements  as  possible  (emphasis  added). 

This  last  relatively  small  technical  detail  got  the  Air  Force  in  trouble  on  the  first  sustained 
protest.  GAO  found  that  the  Air  Force  failed  to  award  appropriate  credit  to  Boeing  for  satisfy¬ 
ing  a  larger  number  of  SRD  requirements  than  Northrop  Grumman/EADS  in  the  five  main 
areas  of  evaluation  laid  out  for  the  key  system  requirements  subfactor.  GAO  recognized  that 
the  Air  Force  had  the  authority  to  select  key  discriminators  among  the  various  requirements, 
as  long  as  the  Air  Force  adhered  to  the  basic  framework  of  the  evaluation  criteria  as  stated  or 
implied  in  the  RFP.  In  addition,  GAO  even  conceded  that  the  Air  Force  could  reasonably  have 
determined  that  larger  differences  between  factors  ranked  lower  in  importance  in  the  RFP 
could  qualitatively  be  determined  during  the  evaluation  to  be  crucial  key  discriminators,  if  a 
reasonable  rationale  was  provided.  But  what  sealed  the  case  for  GAO,  besides  the  violation  of 
the  basic  criteria  presented  in  the  RFP,  was  the  failure  of  the  Air  Force  to  provide  any  clear 
rationale  and  documentation  for  its  final  weighting  and  ranking  of  performance  requirements 
discriminators. 

GAO  provided  several  examples  supporting  its  position.  In  the  aerial  refueling  area,  GAO 
argued  that  Boeing  satisfied  more  requirements  than  the  Northrop  Grumman/EADS  pro¬ 
posal,  but  the  Air  Force  characterized  those  met  by  Boeing  as  being  less-beneficial  overall. 
Indeed,  the  Air  Force  heavily  emphasized  just  one  Northrop  Grumman/EADS  KPP  as  a  major 
discriminator:  exceeding  the  objective  for  fuel  offload  vs.  unrefueled  range.  Indeed,  GAO 
singled  out  this  specific  issue  as  a  separate  major  ground  for  protest  sustainment;  this  separate 
sustained  protest  is  discussed  in  more  detail  below  as  sustainment  number  2.  Flowever,  regard¬ 
ing  the  first  sustained  ground  for  protest,  GAO’s  central  point  was  that,  in  the  aerial  refuel¬ 
ing  area,  most  of  Boeing  discriminators  were  assessed  as  falling  under  KPPs,  whereas  most  of 
Northrop  Grumman/EADS  discriminators  fell  under  the  category  of  non-KPP  and  non-KSA 
requirements.  Nonetheless,  Boeing  did  not  receive  any  credit  for  this. 

GAO  provided  one  more  example.  In  the  operational  utility  area  under  the  key  system 
requirements  subfactor,  the  Air  Force  identified  two  key  discriminators  for  each  proposal. 


52  See  U.S.  Air  Force,  Request  for  Proposal  for  the  KC-X  Tanker  Replacement  Program,  Solicitation  FA8625-07-R-6470, 
Wright-Patterson  AFB,  Ohio:  Aeronautical  Systems  Center,  January  29,  2007,  “Section  M,  Evaluation  Factors  for  Award,” 
§§  2.1  and  2.2.  Also  see  U.S.  Government  Accountability  Office,  2008. 
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Yet  Boeing  received  no  special  credit  for  satisfying  more  SRD  requirements  than  Northrop 
Grumman/EADS  (17  vs.  2).53 

Clearly,  the  Air  Force  had  concluded  that  the  requirements  satisfied  by  Northrop 
Grumman/EADS,  and  the  degree  to  which  the  Northrop  Grumman/EADS  proposal  exceeded 
the  Boeing  design  capabilities  in  certain  requirements  areas,  outweighed  the  fact  that  overall 
the  Boeing  design  satisfied  a  larger  number  of  total  requirements.  This  is  not  surprising,  since 
many  of  the  requirements  dealt  with  relatively  minor  issues.  The  Air  Force  argued  that  the 
SSET  and  SSAC  weighed  all  requirements,  selected  discriminators  based  on  the  best  overall 
value  to  the  Air  Force,  and  ultimately  made  a  subjective  overall  judgment  on  which  proposal 
included  the  most  important  bundle  of  factors  and  provided  the  best  value  to  the  Air  Force. 

As  noted  above,  GAO  concluded  that  in  doing  this  the  Air  Force  made  two  key  mistakes. 
First,  it  failed  to  follow  the  clearly  stated  evaluation  criterion  that  each  offeror  should  try  to 
satisfy  as  many  requirements  as  possible  beyond  the  KPPs,  including  KSAs  and  non-KSAs. 
Boeing  was  awarded  no  credit  for  surpassing  Northrop  Grumman/EADS  in  overall  number  of 
requirements  met.  GAO  pointed  out  that  credit  must  be  given  for  following  the  rankings  and 
priorities  in  the  RFP.  Second,  although  GAO  accepted  that  the  Air  Force  had  the  authority  to 
more  heavily  weight  larger  differences  between  lower-priority  factors,  it  had  to  reasonably  sup¬ 
port  the  reasons  for  doing  this  and  thoroughly  document  them,  something  GAO  claimed  that 
the  Air  Force  did  not  do.  Therefore  GAO  sustained  Boeing’s  protest  in  this  area. 

We  make  one  basic  observation  regarding  this  sustainment:  Clearly,  problems  existed  in 
the  design  and  evaluation  of  the  KC-X  RFP,  particularly  in  the  area  of  requirements  and  the 
methodology  for  evaluating  the  proposal  against  them.  There  are  three  components  to  the 
problem.  First,  the  RFP  did  not  clearly  identify  key  discriminators  nor  carefully  restrict  trade 
space  to  key  discriminators.  Second,  the  RFP  used  imprecise  language  on  the  weighting  and 
evaluation  methodology  for  items  in  the  trade  space.  And  third,  the  documentation  provided 
by  the  Air  Force  did  not  clearly  link  specific  evaluation  language  to  the  RFP. 

We  recognize  the  challenges  of  fully  laying  out  all  the  requirements,  the  key  discrimina¬ 
tors,  and  their  priorities  this  early  in  the  life  of  a  complex  system  that  may  stay  in  the  inven¬ 
tory  for  many  decades.  In  many  respects,  the  evaluation  process  can  be  legitimately  viewed  as 
an  iterative  learning  process  for  both  the  offerors  and  the  government  side.  Flexibility  for  the 
government  to  permit  learning  and  changing  of  emphasis  in  requirements  and  their  priorities 
would  be  ideal.  Unfortunately,  GAO  and  its  well-established  precedents  make  clarity  a  must, 
as  well  as  rigorous  adherence  to  stated  and  implied  evaluation  criteria.  The  Air  Force  must  take 
into  account  this  basic  reality  when  it  designs  and  executes  its  RFPs. 


33  The  two  Northrop  Grumman/EADS  discriminators  were  (1)  more  fuel  from  a  7,000-ft  runway,  and  (2)  a  longer  ferry 
range. 
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Sustain  #2:  Contrary  to  RFP  Language,  USAF 
Gave  Credit  for  Exceeding  KPP  Threshold 

•  RFP:  “No  consideration  will  be  provided  for 
exceeding  KPP  objectives.”  Evaluators  then 
credited  one  proposal  for  achieving  better 
performance  on  air  refueling  KPP  than  the  other. 

•  USAF:  RFP  as  a  whole  and  common  sense  demand 
that  higher  air  refueling  performance  be  rewarded. 

•  GAO:  That  is  not  what  the  criteria  in  the  RFP  say. 

•  Takeaway:  The  source  selection  failed  to  detect  a 
key  inconsistency  in  RFP  language.  Language  in 
one  place  conditioned  language  in  another  to 
produce  a  clear  direction  in  the  RFP  not  to  reward 
higher  air  refueling  performance. 


GAO  Bid  Protests  in  Air  Force  Source  Selections  -31  - 


The  second  ground  for  protest  sustained  by  GAO  probably  received  the  most  publicity 
and  was  considered  by  many  observers  to  be  the  most  substantive  and  clear-cut  of  all  the  eight 
sustained  grounds  for  protest.  At  the  same  time,  it  was  probably  the  most  frustrating  sustained 
ground  for  protest  for  the  Air  Force.  Many  Air  Force  officials  viewed  this  specific  sustainment 
as  most  clearly  violating  “common  sense”  and  undermining  the  obvious  best  interests  of  the 
Air  Force  and  the  warfighter.  Even  more  frustrating  for  the  Air  Force,  GAO  sustained  on  the 
grounds  of  an  apparent  contradiction  among  a  few  short  sentences  buried  deep  within  widely 
differing  parts  of  a  very  lengthy  RFP. 

However,  the  basic  principle  from  the  GAO  perspective  was  extremely  straightfor¬ 
ward.  From  the  GAO  perspective,  the  Air  Force  awarded  significant  credit  to  the  Northrop 
Grumman/EADS  proposal  in  clear  violation  of  a  statement  regarding  evaluation  criteria  in  the 
RFP.  There  is  no  doubt  that  the  Air  Force  action  violated  the  apparent  meaning  of  the  RFP  cri¬ 
teria  language,  although  the  Air  Force  defended  itself  vigorously  with  both  a  strong  common 
sense  argument  as  well  as  an  interesting  interpretation  of  the  wording  in  the  RFP.  But  GAO’s 
basic  argument  was  simple  and  straightforward  and  clearly  linked  to  the  first  sustained  ground 
for  protest. 

This  is  what  happened.  As  noted  in  the  discussion  of  the  first  sustainment,  the  Air  Force 
ended  up  selecting  key  discriminators  among  the  SRD  requirements  grouped  in  five  areas, 
with  a  particular  emphasis  on  aerial  refueling,  operational  utility,  and  airlift  capability.  Propos¬ 
als  had  to  meet  all  KPP  threshold  values,  but  the  RFP  explicitly  stated  that  no  credit  would  be 
awarded  for  exceeding  a  KPP  objective.  The  RFP  laid  out  nine  KPPs.  The  second  KPP  was  fuel 
offload  and  range,  which  was  defined  using  a  chart  equivalent  to  the  KC-135  capabilities.  The 
fuel  offload  vs.  unrefueled  range  KPP  threshold  required  equaling  the  fuel  offload  capabilities 
of  the  KC-135  at  various  unrefueled  ranges  as  indicated  on  the  chart.  The  KPP  objective  was 
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merely  to  exceed  the  threshold.  Both  the  Boeing  and  the  Northrop  Grumman/EADS  proposals 
met  and  exceeded  the  threshold,  and  thus  both  also  met  the  objective.  However,  the  KC-45 
offload  capability  significantly  exceeded  that  of  the  Boeing  KC-767.  As  a  result,  fuel  offload 
at  unrefueled  range  ultimately  became  one  of  the  most  important  discriminators  for  the  Air 
Force.  Common  sense  suggests  that  this  should  not  be  surprising,  given  the  basic  mission  of 
an  aerial  refueling  tanker. 

The  language  in  the  RFP  failed  to  reflect  this  common-sense  interpretation.  The  RFP 
stated  that,  if  there  was  no  objective  stated  for  an  attribute,  the  Air  Force  could  determine 
the  value  of  exceeding  the  threshold  capability  after  the  fact  during  the  evaluation.  But  that 
was  irrelevant  here,  since  an  objective  was  stated,  and  that  was  merely  exceeding  the  threshold 
value. 

The  Air  Force  and  Northrop  Grumman/EADS  argued  unsuccessfully  that  the  basic  mis¬ 
sion  of  an  aerial  tanker,  the  sense  of  the  RFP  as  a  whole,  well-known  Air  Force  policy,  and 
common  sense  all  supported  the  Air  Force  position.54  The  Air  Force  maintained,  not  without 
some  justification,  that  it  was  totally  unreasonable  to  assume  that  it  would  treat  as  equivalent 
“one  drop  more”  of  fuel  versus  thousands  of  pounds  of  additional  fuel  offload  capability  at 
specific  unrefueled  ranges.  In  addition,  the  Air  Force  argued  that  Boeing  should  have  become 
aware  of  the  Air  Force  interpretation  of  this  KPP  during  interim  source  selection  briefings  to 
the  offerors.55 

GAO’s  basic  arguments,  however,  were  eminently  logical  and  difficult  to  counter.  GAO 
maintained  that  fairness  in  competitions  requires  consistent  and  reasonable  adherence  to  selec¬ 
tion  criteria  in  the  RFP.  In  addition,  GAO  noted  that  for  some  other  non-KPP  factors,  the  RFP 
had  clearly  indicated  that  credit  would  be  awarded  for  exceeding  the  objective  capability,  but 
that  this  was  not  the  case  here.  GAO  rightly  pointed  out  that  Boeing  could  reasonably  argue 
that  the  apparent  explicit  denial  of  credit  for  exceeding  the  objective  in  this  case  was  a  con¬ 
scious  Air  Force  effort  to  bound  the  trade  space.  That  is,  it  could  be  viewed  as  an  attempt  by 
the  Air  Force  to  ensure  that  it  would  not  have  to  pay  for  a  more  expensive  design  that  provided 
unwanted  capability. 

In  addition,  it  cannot  be  denied  that  Boeing’s  argument  of  materiality  was  also  reason¬ 
able.  Had  the  RFP  explicitly  offered  credit  for  exceeding  the  fuel  offload  KPP  objective,  Boeing 
might  have  been  motivated  to  submit  a  proposal  based  on  one  of  its  larger  airliners  such  as  the 
Boeing  777.56  Finally,  GAO  argued  that  the  documentation  available  for  the  interim  source 


54  In  addition,  the  Air  Force  argued  that  the  Boeing  protest  had  been  untimely. 

55  Several  credible  senior  Air  Force  officials  we  interviewed  claimed  that  Boeing  was  never  misled  by  the  fuel  offload 
requirement.  According  to  Air  Force  officials,  the  protest  was  the  first  time  Boeing  expressed  the  view  that  the  threshold 
and  objective  offload  requirements  were  essentially  the  same.  In  the  protest,  Boeing  made  the  argument,  successfully,  that 
any  amount  of  fuel  beyond  threshold  met  the  objective  requirement.  In  the  protest,  according  to  senior  Air  Force  officials, 
the  Air  Force  asked  GAO  for  discovery  of  Boeing  internal  documents  that  would  have  shown  how  Boeing  interpreted  this 
requirement  at  the  time  it  prepared  its  proposal.  GAO  denied  the  Air  Force’s  request.  So  the  Air  Force  was  left  with  no  evi¬ 
dence  to  refute  the  protest  argument. 

5^  Some  observers  in  and  outside  the  Air  Force  rejected  this  notion  out  of  hand,  arguing  that  a  variety  of  technical,  cost, 
and  commercial  business  reasons  made  it  unlikely  that  Boeing  would  have  offered  a  777-based  aerial  refueling  tanker  pro¬ 
posal  under  most  realistic  circumstances.  Whereas  Boeing  always  publicly  claimed  it  would  offer  a  KC-777  if  the  Air  Force 
desired  a  larger  tanker,  many  observers  noted  that,  in  addition  to  manufacturing  resource  management  and  scheduling 
issues  related  to  the  much  higher  commercial  sales  of  the  777  airliner,  and  higher  development  and  manufacturing  costs  for 
a  KC-777  tanker,  Boeing  much  preferred  to  offer  a  KC-767  tanker,  a  version  of  which  had  already  been  developed.  See,  for 
example,  Gates,  2009. 
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selection  briefings  indicates  that  the  Air  Force  had  informed  Boeing  only  that  it  had  met  the 
fuel  offload  KPP  objective,  with  no  explicit  clarifications  of  the  Air  Force  interpretation. 

The  Air  Force  advanced  another  interesting  argument  in  its  defense:  that  it  is  impos¬ 
sible  to  exceed  a  KPP  objective  that  is  defined  as  exceeding  the  threshold.  Exceeding  this  KPP 
objective  implies  an  endless  number  of  possibilities  stretching  out  in  principle  to  infinity,  so 
it  is  impossible  to  exceed  this  objective  (one  cannot  exceed  infinity).  Thus,  both  proposals 
exceeded  the  threshold  and  thus  met  the  objective,  but  the  KC-45  was  not  given  any  additional 
credit  for  exceeding  the  objective  any  more  than  the  KC-767,  because  it  is  impossible  to  exceed 
the  objective.  Rather,  independent  of  the  KPP,  the  Northrop  Grumman/EADS  proposal  was 
awarded  credit,  argued  the  Air  Force,  for  having  significantly  greater  fuel  offload  capability 
than  the  Boeing  design,  not  for  exceeding  the  KPP  objective,  something  that  in  this  case  was 
logically  impossible.  This  argument  was  perhaps  too  clever  by  half  and  was  rejected  out  of  hand 
by  GAO. 

At  the  end  of  the  day,  GAO  stuck  to  its  finding  that  the  Air  Force  had  violated  the  selec¬ 
tion  criteria  in  the  RFP,  and  that  fairness  in  competitions  demands  consistent  and  reasonable 
adherence  to  selection  criteria  in  the  RFP. 

Our  major  lesson  from  this  sustained  protest  is  that  the  Air  Force,  with  all  its  extensive 
scrubbing  and  review  of  the  RFP,  failed  to  detect  a  very  small  but  key  inconsistency  in  the  RFP 
language  buried  in  a  couple  of  sentences  in  totally  different  parts  of  the  RFP.  Clearly  the  Air 
Force  is  going  to  have  to  achieve  even  higher  standards  of  rigor  and  thoroughness  in  reviewing 
and  vetting  RFPs  in  future  high-profile  source  selections. 
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Sustain  #3:  USAF  Failed  to  Document  Basis 
for  Evaluation  of  Compatibility  with  All  Aircraft 

•  RFP:  Any  tanker  must  be  able  to  “refuel  all  current  Air  Force 
fixed-wing  tanker-compatible  receiver  aircraft  in  accordance  with 
current  Air  Force  procedures.”  Documentation  for  at  least  one 
aircraft  was  incomplete  in  winning  proposal;  USAF  witnesses  at 
GAO  hearing  could  not  explain  basis  for  decision  for  certification. 

•  USAF:  Problem  arose  in  both  proposals;  USAF  addressed  it  in 
contractual  agreements  to  resolve  shortfall  in  final  tanker  design. 

•  GAO:  Documentation  never  satisfactorily  explains  how  USAF 
resolved  EN  or  even  shows  that  USAF  understood  the  nature  of 
the  problem. 

•  Takeaway: 

-  USAF  failed  to  maintain  adequate  control  of  documentation 
generated  by  EN. 

-  Poor  USAF  testimony  in  hearing  led  GAO  to  collect  additional 
information  that  raised  further  doubts  about  USAF 
understanding  of  shortfall. 
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In  the  third  sustained  ground  for  protest,  GAO  concluded  that  the  Air  Force  assess¬ 
ment  that  the  Northrop  Grumman/EADS  proposal  could  refuel  all  Air  Force  aircraft  under 
current  procedures  was  not  supported  by  the  source  selection  documentation.  This  was  a  key 
shortcoming,  in  that  the  number  one  KPP  threshold  requirement  evaluated  under  the  aerial 
refueling  area  of  the  key  system  requirement  subfactor  necessitated  this  capability.  In  principle, 
if  a  proposal  failed  to  satisfy  this  KPP  threshold  requirement,  it  was  automatically  disqualified 
from  the  competition. 

This  protest  involved  very  technical  operational  issues  and  was  apparently  not  handled 
optimally  by  the  Air  Force  during  the  protest  review  and  hearing  process.  GAO  initially  sus¬ 
tained  the  Boeing  objection  that  the  Air  Force  evaluation  was  not  reasonable  when  concluding 
that  the  Northrop  Grumman/EADS  proposal  had  adequate  “overrun”  and  “break  away”  capa¬ 
bilities  using  current  procedures  for  all  current  Air  Force  fixed-wing  receiver  aircraft.  GAO 
noted  that  current  Air  Force  procedures  require  “overrun”  and  “break  away”  capabilities  under 
certain  specialized  conditions.57 

During  the  proposal  evaluation  period,  the  Air  Force  notified  Northrop  Grumman/ 
EADS  twice  that  the  KC-45  top  speed  was  inadequate  for  the  necessary  capabilities.  The 
KC-45  contractors  initially  recommended  that  the  Air  Force  change  its  procedures  then  later 
proposed  solutions.  However,  GAO  identified  several  unresolved  issues.  These  included  the 
KC-45  dive  speed  exceeding  the  maximum  operating  air  speed;  schedule  and  cost  implications 
of  the  proposed  contractor  solution;  and  airspeed  limitations  that  could  prevent  break  away 
procedures  for  one  specific  Air  Force  receiver  aircraft. 


57  “Overrun”  and  “break  away”  capabilities  refer  to  maneuver,  acceleration,  and  speed  performance  considerations  during 
emergency  procedures  while  refueling  receiver  aircraft. 
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According  to  GAO,  neither  the  available  documentation  nor  hearing  testimony  proved 
that  the  Air  Force  understood  the  qualifications  to  Northrop  Grumman/EADS’s  responses  or 
provided  a  reasonable  basis  for  assuming  that  the  KC-45  could  eventually  fully  comply  with 
existing  procedures. 

Some  Air  Force  personnel  involved  in  this  aspect  of  the  protest  were  dismissive  of  GAO 
claims.  They  claimed  that  the  Air  Force  expressed  similar  concerns  regarding  the  Boeing 
design  proposal  during  the  source  selection.  They  noted  that  Northrop  Grumman/EADS  as 
well  as  Boeing  proposed  solutions  to  this  issue  (which  in  the  case  of  the  KC-45  largely  required 
adjustments  to  software),  and  both  committed  contractually  to  implementing  their  proposed 
solutions  during  the  full-scale  SDD  phase  of  the  program.  The  Air  Force  further  claimed 
that  Northrop  Grumman/EADS  supplied  relevant  flight  test  data  and  documentation  based 
on  actual  A330  flight  test  data  to  support  its  position,  whereas  Boeing  could  not  supply  such 
data,  since  the  specific  proposed  Boeing  design  had  never  flown.  The  Air  Force  also  alleged 
that  GAO  had  incorrectly  based  its  findings  on  Air  Force  interim  assessments.  In  short,  the 
Air  Force  insisted  that  Northrop  Grumman/EADS  had  provided  full  and  reasonable  docu¬ 
mentation  of  solutions  to  the  technical  problem  and  had  satisfied  possible  concerns  about  the 
schedule  and  cost  implications  of  implementing  the  solutions. 

GAO,  however,  rejected  these  arguments,  basically  on  the  grounds  that  adequate  and 
convincing  documentation  in  support  of  the  Air  Force  arguments  was  not  made  available.  In 
addition,  GAO  claimed  that  the  Air  Force  expert  testimony  presented  at  the  protest  hearing 
was  unsatisfactory,  indicating  a  lack  of  awareness  of  the  relationship  between  maximum  oper¬ 
ating  speed  and  design  dive  speed.  Finally,  GAO  argued  that  it  could  not  find  any  documenta¬ 
tion  supporting  the  sufficiency  of  Northrop  Grumman/EADS  proposed  air  speed  capability  to 
perform  the  break  away  maneuver  for  one  specific  Air  Force  receiver  aircraft.58 

In  summary,  GAO  concluded  that  the  Air  Force  assessment  that  the  Northrop 
Grumman/EADS  proposal  could  refuel  all  Air  Force  receiver  aircraft  under  current  proce¬ 
dures  was  not  supported  by  the  documentation.  Since  this  was  a  major  KPP  threshold,  GAO 
noted  that  technically  this  point  alone  disqualified  the  Northrop  Grumman/EADS  proposal 
from  further  consideration. 

The  Air  Force  admitted  that  it  may  not  have  completed  the  paperwork  to  document  the 
full  close-out  of  the  ENs  to  Northrop  Grumman/EADS  regarding  this  issue  but  absolutely 
insisted  that  the  issue  had  been  resolved  to  its  satisfaction  and  that  this  ground  for  protest  sus¬ 
tainment  had  no  merit.  Nonetheless,  it  appears  that  the  GAO  decision  was  fully  legitimate  in 
a  narrow  technical  sense.  Apparently  the  Air  Force  had  not  properly  documented  the  close-out 
of  the  relevant  ENs.  Clearly  the  Air  Force  needs  to  maintain  much  more  careful  and  rigorous 
control  of  the  documentation  generated  by  the  EN  process,  particularly  regarding  the  resolu¬ 
tion  of  specific  issues  and  concerns  raised  to  the  offerors. 

Another  lesson  learned  from  this  sustained  protest  is  that  the  Air  Force  expert  witnesses 
must  be  able  to  respond  convincingly  and  authoritatively  to  aggressive  and  hostile  questioning 
by  the  protester’s  attorneys.  The  failure  of  the  Air  Force  expert  testimony  on  this  issue  to  con¬ 
vince  GAO  that  the  Air  Force  understood  the  technical  issues  and  had  adequately  addressed 
the  problems  raised  additional  doubts  about  the  Air  Force  understanding  of  the  shortfall. 


58  GAO  documents  all  these  points  in  depth  in  its  decision  (U.S.  Government  Accountability  Office,  2008). 
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It  should  be  noted  that  it  is  apparently  no  easy  matter  to  ensure  the  effectiveness  of  wit¬ 
nesses  testifying  at  GAO  hearings.59  According  to  Air  Force  officials,  all  witnesses  were  care¬ 
fully  prepared.  An  Air  Force  attorney  was  assigned  to  work  with  each  Air  Force  KC-X  hearing 
witness  well  in  advance  of  the  hearing.  The  Air  Force  attorney  prepared  the  witnesses  with 
likely  questions  and  made  sure  the  witnesses  were  familiar  with  the  relevant  documents  in  the 
administrative  record.  Before  the  hearing,  the  Air  Force  arranged  for  outside  counsel  from  the 
intervener  to  conduct  a  mock  cross  examination  of  each  witness  under  realistic  conditions. 

Despite  such  thorough  preparations,  the  KC-X  hearings  demonstrated  that  some  indi¬ 
viduals  simply  do  not  do  well  as  witnesses.  They  got  nervous,  they  could  not  communicate  well 
in  that  setting,  or  they  failed  to  give  clear  and  complete  answers.  Thus,  a  major  lesson  learned 
from  KC-X  is  that  the  ability  to  present  information  orally  at  a  protest  hearing  should  be  an 
important  consideration  in  choosing  SSET  leads  in  future  major  procurements. 

Yet,  it  is  also  useful  to  point  out  that,  in  the  view  of  some  Air  Force  officials,  GAO  pri¬ 
marily  uses  this  testimony  to  explain  the  evaluation  process  and  the  administrative  record.  It 
generally  does  not  allow  agencies  to  fill  in  gaps  in  the  administrative  record  through  the  hear¬ 
ing.  Some  senior  Air  Force  officials  believe  that  the  outcome  of  the  aircraft  compatibility  issue 
was  unlikely  to  be  different  regardless  of  who  testified. 


59 


This  discussion  draws  on  the  experience  of  Air  Force  officials  directly  involved  in  the  source  selection. 
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Sustain  #4:  USAF  Treated  Offerors  Differently 
in  Evaluation  of  Operational  Utility 

•  During  discussions,  USAF  decided  not  to  notify  one 
offeror  when  it  downgraded  its  rating  on  an 
operation  utility  KPP,  even  though  it  continued 
discussions  with  the  other  offeror  about  the  same 
KPP. 

•  USAF:  Acquisition  rules  do  not  require  USAF  to  tell 
an  offeror  if  it  downgrades  a  rating  of  a  strength. 

And  the  downgrade  was  ultimately  not  material. 

•  GAO:  GAO  precedent  states  clearly  that  the 
government  must  treat  all  offerors  equally 

•Takeaway:  USAF  seriously  misread  GAO  oversight 
policy 

RAND 
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GAO  sustained  a  fourth  ground  for  protest,  saying  that  the  Air  force  conducted  mislead¬ 
ing  and  unequal  discussions  with  Boeing  regarding  fulfillment  of  KPP  number  seven,  which 
addressed  the  requirements  for  net  ready  capability. 

GAO  describes  the  course  of  events  as  follows.60  During  the  source  selection  evaluation, 
the  Air  Force  identified  concerns  regarding  KPP  number  seven  and  alerted  Boeing  through  an 
EN.  Boeing  responded  with  a  proposed  solution.  The  Air  Force  then  notified  Boeing  that  its 
proposed  solution  met  the  KPP  objective. 

Some  time  later,  when,  according  to  the  program  office,  the  Air  Force  was  reviewing 
all  its  documentation,  one  technical  expert  noticed  that  both  the  Boeing  and  the  Northrop 
Grumman/EADS  proposals  offered  equivalent  capability  in  this  area  but  that  the  two  pro¬ 
posals  were  ranked  differently  against  the  KPP  number  seven  metric.  Specifically,  Northrop 
Grumman/EADS  proposed  approach  to  meeting  the  KPP  number  seven  objective  was  ranked 
as  “partially  met,”  whereas  Boeing’s  approach  with  similar  capability  was  ranked  as  fully  meet¬ 
ing  the  objective.  Since  both  proposals  met  and  exceeded  the  threshold  for  KPP  number  seven, 
and  a  small  change  to  the  ranking  would  have  virtually  no  effect  on  the  outcome  of  the  compe¬ 
tition,  the  program  office  decided  for  the  sake  of  consistency  of  the  evaluation  documentation 
that  Boeing’s  ranking  should  be  changed  to  “partially  met.”  This  change  was  made,  but  Boeing 
was  not  notified  regarding  this  change. 

Later,  as  part  of  its  protest,  Boeing  complained  that  it  had  not  been  treated  fairly  because 
the  change  in  the  ranking  for  KPP  number  seven  had  not  been  communicated  to  it.  Boeing 
further  argued  that  Northrop  Grumman/EADS  had  been  informed  of  their  correct  rating  of 


60  U.S.  Government  Accountability  Office,  2008. 
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“partially  met.”  Most  important,  Boeing  argued  that,  had  they  been  properly  informed  of  the 
change  in  rating,  they  would  have  proposed  a  solution  and  fully  met  the  KPP. 

In  the  view  of  KC-X  program  officials  and  program  counsel,  the  decision  not  to  reopen 
discussions  with  the  protester  over  this  downgrade  was  clearly  within  the  contracting  officer’s 
discretion,  per  FAR  15. 306. 61  In  addition,  they  believed  that  there  would  have  been  no  preju¬ 
dice  from  this  decision,  even  if  it  were  erroneous. 

Many  knowledgeable  observers  in  the  Air  Force  and  industry  believed  that  this  protest 
issue  bordered  on  the  frivolous.  The  Air  Force  argued  that  it  was  not  obligated  to  inform 
Boeing  of  the  changed  rating,  because  this  element  was  part  of  the  trade  space  and,  even  after 
the  change  in  rating,  Boeing’s  proposal  was  still  ranked  with  a  strength  in  this  area,  rather 
than  having  been  changed  to  a  deficiency.  Second,  the  Air  Force  maintained  that  this  error  in 
ranking  and  subsequent  change  had  taken  place  after  all  discussions  with  the  offerors  had  been 
closed,  so  that  it  was  not  appropriate  to  hold  discussions  with  Boeing  over  the  change.  Finally, 
and  perhaps  most  important,  the  Air  Force  claimed  that  the  change  had  absolutely  no  material 
effect  on  the  outcome  of  the  competition. 

GAO  countered  the  Air  Force  arguments  with  three  basic  points.  First,  GAO  reiterated 
the  basic  point  that  whatever  the  reason  or  context,  the  Air  Force  had  misled  Boeing  and 
treated  it  unequally  in  this  case.  Second,  GAO  claimed  the  Air  Force  did  reopen  discussions 
with  the  offerors  on  other  topics  after  FPR,  so  why  not  on  this  topic?  Finally,  GAO  insisted  that 
this  error  could  have  reasonably  prejudiced  Boeing’s  proposal,  since  it  involved  the  only  KPP 
objective  assessed  in  the  operational  utility  area  under  the  key  system  requirements  subfactor. 

GAO  made  it  clear  that  GAO  precedent  clearly  states  that  the  government  must  scrupu¬ 
lously  treat  all  offerors  equally.  In  the  case  of  this  protest,  the  Air  Force  unquestionably  misread 
GAO’s  oversight  policy  and  determination  to  apply  strict  interpretations  of  precedent. 


61  According  to  one  Air  Force  official,  Boeing  certainly  should  have  been  notified  and  given  an  opportunity  to  revise  its 
proposal.  However,  this  official  argues  that  the  relevant  FAR  section  seemed  clear  to  the  evaluators  at  the  time  and  that,  in 
their  view,  did  not  require  such  actions.  The  relevant  FAR  language  states:  “The  contracting  officer  must  .  .  .  discuss  with 
each  offeror  .  .  .  deficiencies,  [and]  significant  weaknesses.  The  contracting  officer  also  is  encouraged  to  discuss  other  aspects 
of  the  offeror’s  proposal  that  could,  in  the  opinion  of  the  contracting  officer,  be  altered  or  explained  to  enhance  materially 
the  proposal’s  potential  for  award.  However,  the  contracting  officer  is  not  required  to  discuss  every  area  where  the  proposal 
could  be  improved.  The  scope  and  extent  of  discussions  are  a  matter  of  contracting  officer  judgment”  [FAR  15.306(d)(3)]. 
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Sustain  #5:  USAF  Failed  to  Ensure  N-G  Acceptance 
of  Organic  Depot  Support  Schedule  Requirement 

•  RFP:  Offeror  will  “plan  and  support  the  agency  to 
achieve  initial  organic  depot-level  maintenance 
within  2  years  after  delivery  of  the  first  full-rate 
production  aircraft.” 

•  USAF:  Despite  N-G’s  rebuff  of  repeated  USAF 
queries  about  this,  the  source  selection  advisory 
council  (SSAC)  and  source  selection  authority 
(SSA)  rules  N-G’s  failure  to  include  this  in  its 
proposal  to  be  an  “administrative  oversight.” 

•  GAO:  USAF  unreasonably  overlooked  failure  to 
comply  with  a  material  RFP  requirement. 

•Takeaway:  Source  selection  did  not  fully  prepare 
the  SSAC  and  SSA  to  evaluate  the  RFP. 
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GAO  determined  with  respect  to  sustainment  number  five  that  the  Air  Force  improperly 
assessed  Northrop  Grumman/EADS  failure  to  commit  to  establishing  organic  logistics,  main¬ 
tenance,  and  repair  support  for  the  KC-X  within  a  two-year  time  frame  as  required  by  the  RFP 
under  the  product  support  subfactor. 

According  to  GAO’s  interpretation,  the  statement  of  objectives  for  the  KC-X  SDD  spe¬ 
cifically  required  that  the  offerors  commit  to  planning  and  supporting  the  establishment  of 
organic  Air  Force  support  capability  for  the  KC-X  within  two  years  of  the  delivery  of  the  first 
full-rate  production  aircraft. 

The  original  Northrop  Grumman/EADS  proposal  did  not  clearly  commit  to  this  two- 
year  time  frame.  The  Air  Force  raised  this  issue  twice  with  the  contractor  through  ENs.  The 
Air  Force  did  not  receive  a  fully  satisfactory  response,  and  eventually  the  SSET  assigned  a 
“weakness”  rating  to  Northrop  Grumman/EADS  in  this  area.  In  its  final  proposal,  Northrop 
Grumman/EADS  noted  this  issue  and  stated  that  it  would  be  resolved  “at  contract  award.”62 
On  reviewing  this  issue,  the  SSAC  challenged  the  SSET  assessment  and  concluded  that 
Northrop  Grumman/EADS  failure  to  explicitly  commit  was  the  result  of  an  “administrative 
documentation  oversight.”63  Apparently  the  SSA  agreed  with  this  assessment. 

After  reviewing  this  issue,  GAO  concluded  that  the  organic  repair  requirement  was  a 
material  requirement,  which  had  been  consciously  rejected  by  Northrop  Grumman/EADS.  In 
GAO’s  interpretation,  this  behavior  technically  disqualified  the  entire  Northrop  Grumman/ 
EADS  proposal. 


62  U.S.  Government  Accountability  Office,  2008,  p.  52. 

63  U.S.  Government  Accountability  Office,  2008,  p.  52. 
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The  Air  Force  argued  that  Northrop  Grumman/EADS  had  clearly  committed  to  provid¬ 
ing  the  required  organic  capabilities  and  that  the  cost  and  schedule  documentation  provided 
by  the  offeror  was  consistent  with  a  two-year  time  frame. 

GAO  forcefully  countered  that  its  review  of  the  offeror’s  schedule  and  cost  documenta¬ 
tion  raised  serious  questions  about  the  Air  Force  assessment  regarding  Northrop  Grumman/ 
EADS  commitment  to  the  two-year  time  frame.  Furthermore,  GAO  claimed  that  both  the 
Air  Force  and  the  offeror  admitted  during  GAO’s  review  process  that  this  shortcoming  was 
intentional. 

Our  conclusion  from  this  episode  is  that  once  again  the  Air  Force  failed  to  adequately 
document  why  Northrop  Grumman’s  logistics,  maintenance,  and  repair  support  for  the  KC-X 
was  acceptable  within  the  two-year  time  frame  required  by  the  RFP.  Neither  the  SSAC  nor 
SSA  recognized  the  deficiency. 
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Sustain  #6:  USAF  Used  “Unreasonable”  Method  to 
Calculate  Military  Construction  Costs 

•  RFP:  Calculation  of  most  probable  life  cycle  cost 
will  include  estimates  of  military  construction 
(MILCON)  costs  implied  by  each  proposal 

•  GAO:  (1)  Evaluation  did  not  account  for  the 
offerors’  specific  proposals.  (2)  USAF  did  not 
reasonably  support  basis  for  the  hypothetical  plan 
it  used  to  calculate  MILCON  costs. 

•  Takeaway: 

-  Evaluation  must  be  consistent  with  RFP 
language 

-  USAF  must  explain  costing  methods  in  enough 
detail  to  justify  the  key  assumptions  it  chooses 
to  make  when  applying  these  methods.  GAO 
will  not  defer  to  the  USAF  on  the  sufficiency  of 

rand  t*iese  assumptions. 

GAO  Bid  Protests  in  Air  Force  Source  Selections  -35  - 


The  sixth  sustained  ground  for  protest  was  one  of  the  most  complex  issues  raised  in 
the  protest,  but  the  underlying  principle  once  again  was  very  simple  and  straightforward. 
Here,  GAO  identified  two  basic  issues.  GAO  concluded  that  the  Air  Force  evaluation  of  the 
Northrop  Grumman/EADS  proposal’s  estimate  of  MILCON  costs  was  unreasonable  and  that 
that  evaluation  was  inconsistent  with  the  evaluation  criteria  in  the  RFP. 

The  first  criticism  here  was  similar  in  some  respects  to  the  criticism  of  the  cost  methodol¬ 
ogy  directed  at  the  Air  Force  regarding  the  CSAR-X  MPLCC  estimate  evaluations.  As  the  pro¬ 
testers  had  claimed  in  the  case  of  the  CSAR-X  sustained  protests,  Boeing  argued  that  the  RFP 
had  requested  detailed  data  derived  from  the  unique  attributes  of  the  specific  aircraft  designs 
that  the  offerors  used  in  their  proposals.  The  Air  Force  had  then  disregarded  the  specifics  of 
each  offeror’s  proposal  when  calculating  the  MILCON  costs  for  each  proposal  design  as  part 
of  the  MPLCC  estimate.  Consequently,  the  Air  Force  assessment  of  MILCON  costs  for  both 
offerors  was  not  consistent  with  the  evaluation  criteria  as  laid  out  in  the  RFP. 

The  RFP  had  required,  and  the  offerors  had  provided,  very  detailed  proposal-specific 
data  for  beddown  of  their  proposed  aircraft  at  a  specific  location,  Fairchild  Air  Force  Base.  But 
according  to  GAO,  the  Air  Force  surveyed  four  bases  for  available  facilities  and  other  physical 
support  attributes,  then  extrapolated  the  results  from  one  of  the  bases  surveyed  to  a  total  of  six 
representative  bases  that  could  be  used  for  beddown  of  the  KC-X.  The  Air  Force  rationale  and 
methodology  for  selecting  these  specific  bases  was  unclear  from  the  GAO  perspective.  More 
important,  the  Air  Force  conducted  this  exercise  before  receiving  the  contractor  proposals. 
While  using  the  basic  proposal  aircraft  dimensions,  the  Air  Force  analysis  ignored  the  specific 
technical  details  provided  by  the  offerors  in  their  proposals  and  never  updated  the  analysis  car¬ 
ried  out  before  receiving  the  final  contractor  proposals. 
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Specific  examples  provided  by  GAO  included  issues  such  as  necessary  seat  storage  space 
and  battery  maintenance  and  repair  facilities.64  GAO  noted  that  the  RFP  clearly  implied  that 
the  data  provided  by  the  offerors  would  be  used  to  assist  in  calculating  MILCON  costs  and 
that  no  other  logical  purpose  appeared  to  exist  for  requesting  the  data  from  the  offerors.  Con¬ 
sequently,  GAO  agreed  with  the  protester  that  the  Air  Force  assessment  of  MILCON  costs  for 
both  offerors  was  not  consistent  with  the  evaluation  criteria  as  laid  out  in  the  RFP.  This  find¬ 
ing  of  course  was  very  similar  in  spirit  to  the  first  sustained  protest  in  the  CSAR-X  program. 
As  in  the  case  of  the  CSAR-X,  GAO  found  that  dissimilar  aircraft  of  much  different  sizes  were 
assigned  identical  MILCON  costs  as  part  of  their  MPLCC.  From  the  GAO  perspective,  this 
appeared  totally  contrary  to  the  evaluation  criteria.65 

This  specific  KC-X  protest  sustainment,  however,  had  a  second  key  element,  which  went 
well  beyond  the  issue  of  inconsistency  with  the  RFP  evaluation  criteria.  GAO  also  found  that 
the  notional  methodology  used  by  the  Air  Force  for  assessing  MILCON  costs  was  not  reason¬ 
able  in  and  of  itself  because  no  documentation  existed  justifying  the  assumptions  behind  the 
approach  adopted.  As  noted  above,  the  Air  Force  took  the  results  of  one  of  the  airbases  out  of 
the  four  it  had  surveyed  before  receiving  the  contractor  proposals  and  extrapolated  these  results 
to  six  other  unspecified  bases.  Two  of  these  bases  were  air  reserve  command  bases  and  two 
were  main  operating  bases  located  outside  the  continental  United  States  (OCONUS).  The  Air 
Force  added  an  arbitrary  percentage  of  increased  costs  to  the  MILCON  cost  estimates  for  the 
OCONUS  bases  to  compensate  for  assumed  higher  costs  of  overseas  MILCON. 

GAO  commented  that  there  was  little  if  any  documentation  of  the  rationale  or  justifica¬ 
tion  for  this  methodology  or  the  rationale  behind  the  approach.  GAO  assumed  that  the  bases 
were  picked  because  they  accommodated  a  similar  number  of  aircraft  as  was  currently  in  the 
plan  for  the  KC-X  beddowns.  However,  the  actual  beddown  sites  for  the  KC-X  have  not  been 
determined,  and  no  real  rationale  was  documented  by  the  Air  Force  to  GAO’s  satisfaction 
for  the  selection  of  the  specific  bases  used  or  for  other  aspects  and  assumptions  used  in  the 
approach  to  estimating  MILCON  costs. 

The  Air  Force  defended  its  approach  by  claiming  that  the  detailed  support  data  that  had 
been  requested  from  the  offerors  were  required  for  the  evaluation  of  the  product  support  factor, 
not  for  calculating  MPLCC.  Furthermore,  the  Air  Force  maintained  that  calculating  detailed 
MILCON  cost  estimates  for  each  aircraft  based  on  the  unique  attributes  of  the  offerors’  pro¬ 
posals  would  not  have  substantially  changed  the  final  MPLCC  comparison.  Finally,  regarding 
an  error  detected  in  the  original  MPLCC  estimates  that  changed  the  lowest  MPLCC  bidder, 
the  Air  Force  noted  that  the  two  offerors  were  still  essentially  equal  in  overall  MPLCC  and  that 
this  error  had  no  material  effect  on  the  outcome  of  the  competition. 

In  response,  GAO  once  again  highlighted  two  basic  principles:  The  Air  Force  must 
(1)  follow  the  evaluation  criteria  laid  out  in  the  RFP  and  (2)  clearly  document  the  assump- 


64  The  aircraft  were  expected  to  be  convertible  from  an  aerial  tanker  configuration  to  both  a  cargo  and  passenger  configu¬ 
ration.  When  configured  as  a  tanker  or  cargo  aircraft,  the  seats  would  be  removed  and  would  have  to  be  stored  at  the  home 
base.  In  the  case  of  batteries,  it  was  not  known  what  type  of  batteries  would  be  used  on  the  KC-45  and  thus  what  sort  of 
battery  supply,  storage,  and  support  facilities  might  be  required.  In  the  case  of  the  767  tanker,  it  was  assumed  that  the  bat¬ 
teries  would  be  similar  to  those  used  on  an  existing  Air  Force  aircraft,  which  then  could  share  facilities  already  in  place  to 
support  battery  storage,  maintenance,  and  repair. 

65  GAO  also  noted  that  an  Air  Force  mistake  was  detected  that  actually  changed  the  lowest  MPLCC  candidate  in  the  com¬ 
petition.  However,  it  was  noted  that  the  overall  MPLCC  for  both  aircraft  was  still  so  close  that  cost  considerations  played 
a  very  minor  role  in  determining  the  winner. 
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tions  and  approach  it  uses  to  calculate  MILCON  costs.  GAO  raised  this  first  point  in  the 
first  CSAR-X  protest  as  well  as  the  first  and  second  sustained  grounds  for  protest  in  the  KC-X 
source  selection.  The  second  point  is  perhaps  more  surprising.  By  arguing  that  the  Air  Force 
must  carefully  and  thoroughly  document  the  methodology  and  key  assumptions  it  uses  in  its 
evaluations  of  proposal  costs,  the  GAO  decision  demonstrated  that  it  will  not  defer  to  the  Air 
Force  on  the  sufficiency  of  these  assumptions  without  extensive  documentation. 
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Sustain  #7:  USAF  Inappropriately  Added  Costs 
for  Nonrecurring  Engineering  Costs 

•  If  a  cost  estimate  is  deemed  “unrealistically  low,” 

USAF  procurement  policy  allows  evaluators  to 
increase  the  cost  estimate  before  comparing  offers. 

•  USAF:  Boeing  claimed  the  price  it  would  pay  for 
commercial  aircraft  covered  development  costs  and 
so  provided  no  cost  data  to  justify  price.  This  made 
Boeing’s  cost  estimate  risky. 

•  GAO:  Presence  of  risk  is  not  sufficient  to  justify 
adding  cost.  To  add  cost,  USAF  must  justify  its 
judgment  that  the  cost  estimate  is  low.  To  do  this, 

USAF  would  be  allowed  to  demand  relevant  cost 
data. 

•Takeaway:  USAF  did  not  understand  GAO  precedent 
relevant  to  adjustment  of  unrealistic  cost  estimates 

RAND 
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In  GAO’s  seventh  sustained  ground  for  protest,  it  played  an  activist  role  we  did  not  see 
in  any  of  the  sustainments  discussed  above — or  in  any  other  decision  we  examined  over  the 
course  of  this  study.  It  sustained  a  ground  for  protest  that  even  the  protester  had  not  raised 
in  its  protest.  Rather,  GAO  originated  a  ground  for  protest  of  its  own,  arguing  that  the  Air 
Force  cannot  adjust  an  MPLCC  estimate  for  cost  risk — even  if  the  Air  Force  has  legitimate 
grounds  for  being  uncertain  about  an  offeror’s  MPLCC  estimate — without  first  conducting  its 
own  MPLCC  cost  estimation  based  on  an  offeror’s  unique  methods,  materials,  and  approach 
and  justifying  an  alternative  estimate  higher  than  the  offeror’s  estimate.  Because  the  Air  Force 
adjusted  Boeing’s  estimate  of  its  MPLCC  upward  without  generating  such  an  estimate  of  its 
own,  GAO  ruled  that  the  adjustment  was  unreasonable.  In  this  ruling,  GAO  in  effect  sus¬ 
tained  a  ground  for  protest  not  contained  in  Boeing’s  protest.  How  did  this  happen? 

Boeing’s  KC-X  proposed  solution  was  a  derivative  of  its  baseline  B.767-300  commercial 
airliner  built  by  Boeing  Commercial  Airplanes  (BCA),  a  division  of  the  Boeing  Company. 
Boeing’s  KC-X  proposal  stated  that  it  planned  to  purchase  its  baseline  767  aircraft  from  BCA 
as  commercial  items  using  a  fixed-price  (FP)  contract.  Furthermore,  Boeing  maintained  that 
this  FP  contract  would  include  nonrecurring  engineering  costs  and  that  the  Boeing  military 
aircraft  division  would  not  have  access  to  these  proprietary  data.  Boeing  noted  that  Part  12 
of  the  FAR66  discourages  the  government  from  requesting  cost  data  for  commercial  items  and 
forbids  the  imposition  of  a  requirement  for  government-certified  cost  data. 

The  Air  Force  was  uneasy  with  Boeing’s  initial  refusal  to  provide  more  detailed  nonrecur¬ 
ring  engineering  cost  data,  because  the  Boeing  KC-X  design  proposal  was  a  significant  modi¬ 
fication  and  entailed  a  certain  degree  of  risk.  The  Air  Force  repeatedly  requested  more  detailed 


(,('  FAR  Part  12  covers  the  acquisition  of  commercial  items.  Noncommercial  procurement  regulations  are  covered  largely 
under  FAR  Part  15,  “Contracting  by  Negotiation.” 


Protests  Sustained  in  the  CSAR-X  and  KC-X  Source  Selections  89 


cost  data  from  Boeing  to  substantiate  the  nonrecurring  cost  estimates  provided  by  Boeing  in  its 
proposal.  Boeing  repeatedly  refused  to  provide  this  documentation.  Ultimately,  the  Air  Force 
concluded  that  Boeing’s  nonrecurring  engineering  cost  estimate  clearly  entailed  at  least  mod¬ 
erate  risk  and,  therefore,  lacking  fuller  cost  documentation  from  Boeing,  should  be  adjusted 
upward  to  compensate  for  risk. 

Boeing  protested  this  action  by  the  Air  Force.  It  is  interesting  to  note  that  GAO  rejected 
the  grounds  on  which  Boeing  protested  the  Air  Force  action.  It  rejected  Boeing’s  argument 
against  providing  additional  cost  documentation  based  on  the  plan  for  an  FP  contract  with 
BCA.  GAO  rightly  pointed  out  that  FAR  Part  12  only  prohibits  the  government  from  requir¬ 
ing  government  certified  cost  data  but  allows  the  government  to  request  and  require  additional 
cost  data  when  necessary.  Furthermore,  GAO  pointed  out  that  Boeing  had  not  yet  even  negoti¬ 
ated  an  FP  agreement  with  BCA. 

In  addition,  GAO  agreed  with  the  Air  Force  decision  to  assign  moderate  risk  to  the 
Boeing  cost  estimate  for  nonrecurring  engineering  costs  on  the  grounds  that  Boeing  refused  to 
provide  sufficient  cost  documentation.  However,  at  the  same  time  GAO  rejected  the  Air  Force 
justification  and  rationale  for  its  upward  adjustment  of  Boeing’s  cost  estimate. 

GAO  argued  that,  in  accordance  with  GAO  precedent  and  the  RFP  evaluation  metrics 
that  referenced  FAR  Part  15.404,  the  Air  Force  should  have  conducted  an  analysis  of  cost  real¬ 
ism  to  justify  adjusting  Boeing’s  cost  estimate  upward.67  The  Air  Force  can  add  dollarized  risk 
to  an  MPLCC  estimate  per  the  RFP  only  if  such  an  adjustment  results  in  a  more  probable  cost 
supported  by  a  cost  realism  analysis  that  finds  the  offeror’s  cost  estimate  to  be  unrealistic.  GAO 
found  no  supporting  documentation  or  evidence  that  the  Air  Force  had  analyzed  the  cost  real¬ 
ism  of  the  Boeing  nonrecurring  cost  estimate  and  found  it  unrealistic.  In  what  appears  to  be  a 
very  narrow  reading  of  FAR  Part  15  and  GAO  precedent,  GAO  found  that  the  Air  Force  had 
to  formally  demonstrate  and  fully  document  that  the  Boeing  cost  estimate  was  not  realistic 
before  it  could  justify  adjusting  that  estimate  upward,  even  though  GAO  agreed  with  the  Air 
Force  risk  assessment. 

Given  how  unusual  the  activist  stance  behind  this  sustainment  was,  it  is  hard  to  draw  a 
clear  lesson  from  it.  Perhaps  the  best  lesson  to  draw  is  that  future  protesters  can  raise  the  kind 
of  objection  that  GAO  generated  itself  in  this  sustainment.  To  avoid  such  an  objection,  the 
Air  Force  must  comply  with  very  strictly  drawn  procedural  rules  when  it  dollarizes  risk.  These 
rules  derive  from  broad  policy  stated  in  FAR  Parts  15.305  and  15.404,  implemented  through 
a  very  pointed  GAO  interpretation.  More  broadly,  for  better  or  worse,  this  sustainment  indi¬ 
cates  that  the  Air  Force  did  not  fully  understand  or  appreciate  GAO  precedent  relevant  to  the 
adjustment  of  unrealistic  cost  estimates,  or  the  full  technical  implications  of  that  precedent  for 
the  interpretation  of  the  FAR. 


67  FAR  Part  15.305  (1)  states  that  “when  contracting  on  a  cost-reimbursement  basis,  evaluations  shall  include  a  cost  realism 
analysis  to  determine  what  the  Government  should  realistically  expect  to  pay  for  the  proposed  effort,  the  offeror’s  under¬ 
standing  of  the  work,  and  the  offeror’s  ability  to  perform  the  contract.  .  .  .  The  contracting  officer  shall  document  the  cost  or 
price  evaluation.”  FAR  Part  15-404-l(d)(2)  confirms  that  “cost  realism  analyses  shall  be  performed  on  cost-reimbursement 
contracts  to  determine  the  probable  cost  of  performance  for  each  offeror,  (i)  The  probable  cost  may  differ  from  the  proposed 
cost  and  should  reflect  the  Government’s  best  estimate  of  the  cost  of  any  contract  that  is  most  likely  to  result  from  the 
offeror’s  proposal.  The  probable  cost  shall  be  used  for  purposes  of  evaluation  to  determine  the  best  value,  (ii)  The  probable 
cost  is  determined  by  adjusting  each  offeror’s  proposed  cost,  and  fee  when  appropriate,  to  reflect  any  additions  or  reductions 
in  cost  elements  to  realistic  levels  based  on  the  results  of  the  cost  realism  analysis.”  That  is,  a  cost  realism  analysis  will  be 
conducted  before  adjusting  an  offeror’s  cost  estimate. 
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Sustain  #8:  USAF  Used  “Unreasonable” 
Estimate  of  Nonrecurring  Engineering  Costs 

•  Lacking  cost  data  discussed  under  “Sustain  #7,” 

USAF  devised  a  method  to  evaluate  risk  associated 
with  Boeing’s  nonrecurring  engineering  costs. 

•  GAO:  USAF  never  justified  why  its  chosen  method 
would  yield  a  “reliable  predictor  of  anticipated 
growth  of  the  protester’s  non-recurring  engineering 
costs.” 

•Takeaway  repeats  lesson  from  “Sustain  #6”:  USAF 
must  explain  costing  methods  in  enough  detail  to 
justify  the  key  assumptions  it  chooses  to  make 
when  applying  these  methods.  GAO  will  not  defer 
to  the  USAF  on  the  sufficiency  of  these 
assumptions. 

RAND 
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GAO’s  eighth  sustained  ground  for  protest  on  the  KC-X  program  is  closely  related  to 
the  seventh.  Here  again,  GAO  explicitly  criticized  the  cost  methodology  employed  by  the  Air 
Force.  It  found  that  the  Air  Force  methodology  for  estimating  the  most  likely  nonrecurring 
engineering  costs  for  the  Boeing  proposal  was  flawed,  and  thus  the  resulting  estimate  was  not 
reasonable.  So,  the  seventh  sustainment  ruled  that  the  Air  Force  did  not  use  an  approach  to 
cost  estimation  that  could  (1)  generate  a  better  MPLCC  estimate  than  Boeing  offered  in  its 
proposal  and  (2)  support  clear  documentation  of  the  Air  Force’s  judgment  that  its  estimate  was 
better.  The  eighth  sustainment  ruled  that,  even  if  the  Air  Force  had  used  such  an  approach,  the 
specific  cost  model  it  employed  to  dollarize  cost  risk  would  have  been  inappropriate  to  use  in 
it.  These  fine  distinctions  offer  some  insight  into  why  many  in  the  Air  Force  find  GAO  rulings 
to  be  overly  technical.  They  also  illustrate  how  GAO  hearing  officials  think  and  how  the  Air 
Force  acquisition  personnel  must  learn  to  think  to  improve  Air  Force  experience  with  GAO 
bid  protests  in  the  future. 

As  explained  above,  Boeing  declined  to  provide  the  Air  Force  with  detailed  cost  data  to 
document  the  estimated  cost  of  its  nonrecurring  engineering  work  for  its  proposed  767  tanker. 
The  Air  Force  used  a  probabilistic  Monte  Carlo  simulation  to  calculate  a  percentage  increase 
adjustment  in  Boeing’s  estimate  to  reflect  likely  cost  growth.  As  a  check  on  the  credibility  of 
the  results  of  this  approach,  the  Air  Force  reviewed  the  Boeing  P-8A  Poseidon  Multi-Mission 
Maritime  Aircraft  contract  with  the  Navy  and  found  that  the  cost  growth  on  that  contract  to 
be  similar  to  the  upward  adjustment  suggested  by  the  Monte  Carlo  simulation. 

However,  GAO  concluded  that  this  methodology  was  flawed.  The  data  the  Air  Force  used 
for  the  Monte  Carlo  simulation  came  from  reports  written  by  GAO  and  the  RAND  Corpora- 
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tion.68  These  data  represented  historical  averages  of  overall  program  cost  growth.  They  did  not 
reflect  merely  the  nonrecurring  engineering  cost  portion  of  past  programs  but  rather  applied 
to  entire  program  cost  histories,  including  development  and  production.  In  the  bid  protest, 
GAO  therefore  determined  that  the  data  used  in  the  Monte  Carlo  simulation  were  irrelevant 
for  estimating  cost  growth  for  any  single  element  or  phase  of  a  program,  such  as  nonrecurring 
engineering  costs.  GAO  argued  that  the  cost  realism  evaluation  required  by  FAR  Part  15.404-1 
must  be  reasonably  and  rigorously  conducted  no  matter  what  methodology  is  chosen.  Since 
the  Air  Force  used  data  derived  from  entire  program  histories  to  evaluate  the  cost  realism  of 
the  offeror’s  cost  estimate  for  one  phase  of  the  program,  GAO  concluded  that  the  Air  Force 
approach  was  unreasonable. 

Thus,  the  lesson  for  the  Air  Force  from  the  eighth  sustained  ground  for  protest  is  virtu¬ 
ally  the  same  as  from  the  sixth.  The  Air  Force  must  thoroughly  justify  and  document  the  cost¬ 
estimating  methodology  that  it  uses  to  assess  offerors’  cost  estimates.  Without  such  documen¬ 
tation,  GAO  will  not  defer  to  the  assumptions  made  by  the  Air  Force. 


68  Keep  in  mind  that  GAO  analytic  reports  are  written  on  the  other  side  of  a  Chinese  wall  from  the  GAO  Office  of  Gen¬ 
eral  Counsel  that  reviews  bid  protests.  In  the  context  of  a  specific  bid  protest,  these  two  parts  of  GAO  do  not  communicate 
directly  and  are  essentially  separate  organizations. 
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For  KC-X,  GAO  Denied  Far  More 
Grounds  for  Protest  Than  It  Sustained 

*  GAO  did  not  document  vast  majority  of  grounds  for  protest 
denied 

*  Denied  grounds  documented  addressed  USAF  evaluation  of 

-  Risk  associated  with  system  integration  and  software 

-  Management  risk  associated  with  production  lines  and 
production  approach 

-  Past  performance  risk 

-  The  “IFARA  factor”  measuring  fleet-wide  refueling 
capability 

-  A  variety  of  cost  issues 

*  in  each  of  these  cases,  GAO  found  that  the  USAF  had 
complied  with  federal  rules  and  “deferred  to  USAF  judgment” 
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Now  that  we  have  reviewed  all  eight  of  the  KC-X  grounds  for  protest  sustained  by  GAO, 
some  additional  observations  are  possible.  Perhaps  one  of  the  most  important  contextual  obser¬ 
vations  is  to  keep  in  mind  that,  although  GAO  sustained  eight  grounds  for  protest,  including 
some  that  might  appear  to  various  observers  to  be  highly  technical  and  contrary  in  spirit  to 
the  best  interests  of  the  warfighter,  GAO  in  fact  denied  far  more  grounds  for  protest  than  it 
sustained.  Although  the  exact  count  is  in  dispute,  it  appears  that  Boeing  filed  well  over  100 
discrete  grounds  for  protest  on  the  KC-X  case.  Something  like  90  percent  of  these  were  denied, 
but  GAO  documented  virtually  none  of  the  denied  grounds  for  protest  for  the  public.  Indeed, 
even  the  subject  matter  of  many  of  them  was  not  made  public  in  any  formal  sense. 

However,  GAO  did  provide  some  documentation  on  several  grounds  for  protest  that  it 
denied.69  These  included  such  issues  as  the  following: 

•  Risk  associated  with  system  integration  and  software.  Here,  Boeing  alleged  that  the  Northrop 
Grumman/EADS  proposals  should  have  received  a  higher  risk  rating  than  Boeing’s.  The 
Air  Force  argued  that  both  offerors  proposed  substantial  software  reuse  and,  because  of 
this  and  a  variety  of  other  issues,  the  Air  Force  ranked  both  proposals  as  moderately  risky. 
GAO  agreed  with  the  Air  Force  and  denied  the  Boeing  protest. 

•  Management  risk  associated  with  production  lines  and  production  approach.  Boeing  objected 
to  the  Air  Force  assessment  of  the  production  approach  risk  of  the  Northrop-Boeing/ 
EADS  proposal,  but  GAO  upheld  the  Air  Force  position. 

•  Past  performance  risk.  Boeing  protested  three  aspects  of  the  Air  Force  assessment  of  both 
offerors’  past  performance.  The  Boeing  protests  covered  the  relevance  of  the  contracts 


69  U.S.  Government  Accountability  Office,  2008. 
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selected,  the  alleged  unequal  treatment  of  the  offerors,  and  the  alleged  lack  of  docu¬ 
mentation  of  the  past  performance  judgments.  GAO  noted  that  although  the  two  offer¬ 
ors  received  the  same  overall  rating  in  past  performance,  the  SSA  awarded  Northrop 
Grumman/EADS  a  higher  rating  than  Boeing  in  the  program  management  area.  GAO 
concluded  that,  based  on  the  available  documentation,  the  Air  Force  assessment  and  the 
SSA  decision  were  reasonable. 

•  The  IFARA  factor  measuring  fleet-wide  refueling  capability 7°  Boeing  protested  the  Air 
Force  evaluation  determining  that  the  KC-45  was  superior  to  the  Boeing  proposal  in  the 
IFARA  analysis,  because  the  Air  Force  allegedly  based  this  solely  on  the  fleet  effectiveness 
factor  and  failed  to  evaluate  other  factors  and  insights.  GAO  reviewed  this  analysis  and 
concluded  that  the  SSAC  and  SSA  did  review  other  factors  and  thus  found  that  the  Air 
Force  evaluation  was  reasonable. 

•  Various  cost  issues.  Boeing  protested  a  variety  of  other  cost  evaluations  made  by  the  Air 
Force,  including  upward  adjustments  of  Boeing’s  estimated  costs  for  budgetary  aircraft 
costs  and  O&S  repair  costs  and  the  assessment  of  Northrop  Grumman/EADS  KC-45 
fuel  costs.  GAO  found  nothing  in  the  record  to  sustain  these  protests. 

In  short,  in  these  cases  and  apparently  many  others,  GAO  found  that  the  Air  Force  had 
carefully  followed  government  regulations  and  GAO  precedents  and  deferred  to  the  Air  Force 
on  substantive  judgments. 


70  IFARA  was  an  evaluation  of  fleet  refueling  effectiveness  under  a  variety  of  scenarios  using  an  existing  model  to  develop 
a  final  fleet  effectiveness  value  score. 
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Other  Observations  on  KC-X 

•  Protesters  filed  100+  grounds  for  protest  following  source 
selection  decision. 

-  GAO  denied  all  but  eight,  usually  without  explanation. 

-  “Shot  gun”  approach  seriously  tested  SPO  capacity  to 
respond  to  protests 

-  SPO  had  little  insight  into  which  of  these  to  take  seriously 

•  Sustainments  occurred  despite  intense  quality  control,  quality 
assurance  efforts. 

-  A  closer,  more  technical  scrub  would  be  required  to 
resolve  inconsistencies,  lack  of  discipline 

-  Formal  legal  review  failed  to  anticipate  GAO  concerns 

*  GAO  relied  heavily  on  its  own  judgment  of  reasonableness  of 
cost  estimation.  That  judgment  appears  to  be  well  informed. 

*  Difficult  to  reconcile  GAO’s  materiality  determinations  with 
formal  GAO  definition  of  materiality. 
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This  slide  concludes  with  some  broad  points  regarding  the  overall  Air  Force  experience  on 
the  KC-X  protests  and  draws  some  comparisons  with  the  CSAR-X  protest  experience.  As  noted 
above,  Boeing  filed  more  than  100  grounds  for  protest  following  the  source  selection.  GAO 
denied  all  but  eight,  with  virtually  no  public  explanation  of  the  reasons  for  the  more  than  90 
percent  of  grounds  for  protest  that  were  denied. 

It  is  clear  that,  in  both  the  CSAR-X  and  KC-X  experiences,  the  protesters  used  a  shot  gun 
approach  in  an  attempt  to  overwhelm  the  SPO’s’  capability  to  respond  to  the  protests  and  in 
the  hope  of  finding  at  least  one  point  that  might  gain  traction  with  GAO.  In  both  cases,  the 
protests  nearly  overwhelmed  the  program  offices  with  workload.71 

Boeing  filed  at  least  eight  waves  of  potential  protest  grounds.  By  the  end,  Boeing’s  protest 
filled  1,000  pages.  Strong  evidence  exists  that  it  probably  learned  a  great  deal  about  how  to  do 
this  from  the  CSAR-X  case.  The  KC-X  program  office  was  heavily  challenged  by  the  deluge 
of  protest  grounds  but  apparently  was  not  hit  as  hard  as  the  CSAR-X  program  office,  perhaps 
because  the  KC-X  program  office  had  also  learned  from  the  CSAR-X  experience. 

Perhaps  an  even  more  important  point  is  that,  in  both  the  CSAR-X  and  KC-X  cases, 
GAO  found  errors  and  sustained  a  small  number  of  grounds  for  protests,  despite  intensive  and 
rigorous  quality  control  and  QA  efforts  by  the  program  offices  and  the  Air  Force  acquisition 
professional  staff,  both  before  the  protests  and  during  the  protests.  This  is  a  significant  point, 
because  one  of  the  key  process  improvements  the  Air  Force  has  identified  for  avoiding  future 
protests  is  the  implementation  of  greater  oversight  and  outside  review  of  the  RFP  development 
and  proposal  evaluation  processes.  However,  it  is  difficult  to  imagine  a  more  rigorous  and  sub- 


As  with  much  of  the  material  in  this  chapter,  the  material  in  this  paragraph  and  those  that  follow  is  based  on  interviews 
with  individuals  close  enough  to  the  source  selection  to  base  these  observations  on  their  own  personal  knowledge. 
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stantial  review  process  than  the  one  that  actually  took  place  in  the  cases  of  both  the  CSAR-X 
and  KC-X.  The  entire  process  was  vetted  multiple  times  up  and  down  the  chain  of  command 
and  through  the  use  of  numerous  outside  experts  and  assessment  panels.  High-quality,  formal 
legal  reviews  failed  to  ferret  out  the  issues  that  gained  traction  with  GAO  and  led  to  sustained 
protests.  And  no  review  could  have  caught  problems  in  costing  technique  unless  that  review 
had  challenged  the  adequacy  of  this  technique  at  a  basic  level. 

Perhaps  the  most  effective  course  the  Air  Force  could  adopt  would  be  to  form  extremely 
aggressive  “red  teams,”  possibly  including  high-powered  outside  attorneys,  to  scour  the  RFPs 
and  evaluation  documentation  with  hostile  intent  to  simulate  the  activities  of  a  protesting  con¬ 
tractor.  We  discuss  this  in  more  detail  below. 

In  cases  where  the  Air  Force  did  not  fully  document  its  cost  analysis  approach  and  meth¬ 
odology,  GAO  relied  on  its  own  judgment  of  reasonableness.  In  several  cases,  GAO’s  judgment 
appears  well  informed. 

As  in  the  CSAR-X  case,  officials  in  the  KC-X  program  office  had  great  difficulty  distin¬ 
guishing  serious  grounds  for  protest  from  ones  they  could  easily  brush  away.  They  believe  that 
they  could  not  have  predicted  with  much  confidence  which  would  ultimately  “stick.”  They 
used  the  level  of  interest  GAO  showed  in  specific  issues  to  try  to  predict,  but  GAO  sustained 
some  grounds  for  protests  without  even  asking  about  them  in  hearings. 

On  several  of  the  specific  grounds  for  protest,  real  questions  can  be  raised  regarding  the 
materiality  of  the  issues  reviewed  and  sustained  by  GAO.  Several  of  the  sustained  grounds  for 
protest  do  not  seem  to  meet  GAO’s  standard  of  materiality.  This  issue  and  the  one  summa¬ 
rized  in  the  prior  paragraph  greatly  complicate  the  Air  Force’s  goal  of  writing  better  RFPs  and 
conducting  more  disciplined  source  selection  evaluations,  because  GAO’s  particular  areas  of 
interest  and  focus  sometimes  remain  unclear. 


CHAPTER  FIVE 


Synthesis  of  Findings 


Road  Map 

•  How  source  selections  and  bid  protests  work 

•  Patterns  for  bid  protests  during  FY2000-08 

•  Protests  sustained  in  CSAR-X,  KC-X  source 
selections 

•  Synthesis  of  findings 

•  Potential  changes  in  policy  or  practice 
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This  chapter  synthesizes  findings  from  the  material  we  have  just  discussed  and  from  broader 
discussions  with  knowledgeable  people  inside  and  outside  the  Air  Force  and  GAO.  It  begins 
with  an  overview  of  broad  patterns  in  Air  Force  experiences  with  GAO  bid  protests.  These  pat¬ 
terns  suggest  that  the  Air  Force  should  focus  any  efforts  to  adjust  its  policies  regarding  source 
selection  and  protests  on  acquisitions  that  are  likely  to  attract  offerors  who  could  become 
“sophisticated  protesters.”  It  describes  what  this  means  and  then  reviews  a  series  of  adjustments 
that  the  Air  Force  could  make  to  reduce  the  likelihood  that  sophisticated  protests  succeed  and 
to  mitigate  the  costs  to  the  Air  Force  when  they  do.  These  suggestions  are  generally  consistent 
with  the  Air  Force’s  current  direction  as  defined  in  the  AIP,  but  they  (1)  refine  some  directions, 
and  (2)  offer  some  potential  extensions.1 


1  U.S.  Air  Force,  2009a. 
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Overall  USAF  Experience  with  Bid 
Protests  Has  Been  Positive 

•  During  1991-2008,  the  number  of  protests  against 
USAF  source  selections  fell  fairly  steadily 

•Since  2001,  the  percentage  of  protests  that  led  the 
USAF  to  change  its  source  selection  decisions  has 
dropped  fairly  steadily 

•  During  2000-08,  protests  led  USAF  to  change  only 
0.2%  of  all  source  selection  decisions 

•  Even  in  recent,  highly  visible  problem  cases,  GAO 
has  dismissed  or  denied  over  90%  of  individual 
grounds  for  protest  that  arose  within  those  source 
selections 
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Overall  USAF  Experience  with  Bid  Protests  Has  Been  Positive 

Looked  at  broadly,  the  Air  Force  experience  with  GAO  bid  protests  over  the  last  two  decades 
has  been  positive.  Relative  to  the  total  number  of  contract  awards,  the  total  number  of  pro¬ 
tests  each  year  has  fallen  since  1991.  The  number  going  to  merit  reviews  has  trended  down¬ 
ward.  The  number  of  corrective  actions  rose  somewhat  during  the  1990s  but  has  been  trending 
down  since  2001.  Throughout  the  period,  the  number  of  merit  protests  sustained  by  GAO  has 
remained  very  low,  with  no  apparent  trend.  The  net  result  of  all  these  trends  is  that,  during 
2000-2008,  the  Air  Force  had  to  change  only  about  0.2  percent  of  all  source  selections  associ¬ 
ated  with  contract  awards  in  response  to  protests. 

Somewhat  more  subtly,  the  Air  Force  has  also  displayed  a  high  level  of  success  with  the 
large  number  of  protests  that  have  characterized  recent  sophisticated  efforts  to  overturn  Air 
Force  source  selections.  No  one  maintains  an  official  record  of  how  many  discrete  grounds  for 
protest  occur  within  any  one  acquisition.  Reasonable  estimates  suggest  that  the  first  effort  to 
overturn  the  CSAR-X  source  selection  decision  involved  about  130  issues.  GAO  rejected  the 
protest  grounds  for  all  of  these  but  one.  The  effort  to  overturn  the  KC-X  source  selection  deci¬ 
sion  also  involved  about  130  issues;  GAO  rejected  the  protest  grounds  for  all  of  these  but  eight. 
GAO  only  had  to  sustain  one  issue  in  a  protest  to  force  a  change  in  the  Air  Force  approach. 
But  it  is  worth  keeping  in  mind  how  many  faulty  claims  protesters  made  to  achieve  a  sustain¬ 
ment  on  a  single  issue. 

This  experience  suggests  that,  looked  at  broadly,  Air  Force  source  selection  policies  and 
practices  work.  Improvements  are  always  possible,  but  any  sense  of  crisis  associated  with  recent 
events  should  be  kept  in  perspective.  The  broad  record  suggests  that  far-reaching  changes  in 
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Air  Force  source  selection  policies  and  practices  are  not  required  and,  in  fact,  might  even  lead 
to  more  harm  than  good.  The  Air  Force  should  target  change  where  the  problem  is  largest. 
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USAF  Should  Be  Cautious  About 
Generalizing  from  CSAR-X,  KC-X  Cases 

•  Both  used  preexisting  designs.  It  was  hard  to  define 
requirements  clearly  while  preserving  competition. 

-  Must  define  requirements  generally  enough  to  induce  all 
desired  sources  to  participate 

*  Both  involved  high-value  acquisitions  with  long-term 
implications  for  global  industrial  base 

-  Winners  may  be  last  global  providers  standing 

-  Fiduciary  responsibility  may  require  a  CEO  to  protest  any 
loss,  even  with  a  small  probability  of  reversing  it 

-  Prominence  of  foreign  offerors  raised  the  political 
visibility  of  both  source  selections 
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USAF  Should  Be  Cautious  About  Generalizing  from  CSAR-X,  KC-X  Cases 

Despite  this  broad  record  of  positive  experience,  the  Air  Force  has  drawn  sharp  criticism  for 
the  protests  that  GAO  sustained  in  the  CSAR-X  and  KC-X  programs.  Something  different 
is  clearly  occurring  here.  In  response,  the  Air  Force  undertook  aggressive  efforts  to  adjust  its 
acquisition  policies  and  its  source  selection  policies  in  particular.  But  before  focusing  too  much 
on  changes  in  policies  and  practices  designed  to  address  issues  that  arose  in  these  two  acquisi¬ 
tions,  the  Air  Force  should  recognize  how  unusual  they  are,  relative  to  other  large  Air  Force 
acquisitions. 

At  a  minimum,  with  the  exception  of  two  other  acquisitions  tainted  by  the  personal  scan¬ 
dals  associated  with  Darken  Druyun,  they  were  the  only  Air  Force  system  acquisitions  since 
2001  in  which  GAO  sustained  protests.  Some  even  argue  that  the  Druyun-related  scandals  in 
themselves  invited  GAO  to  view  the  Air  Force  system  acquisitions  that  followed  with  closer 
scrutiny  and  to  hold  the  Air  Force  to  a  higher  standard  than  it  had  in  the  past.  We  found  no 
evidence  of  this.  But  these  two  acquisitions  do  differ  from  other  Air  Force  acquisitions  in  sev¬ 
eral  other  important  ways. 

First,  both  involve  system  designs  likely  to  be  derived  from  preexisting  designs.  The 
CSAR-X  design  was  expected  to  draw  heavily  on  the  design  of  preexisting  military  and  com¬ 
mercial  helicopters.  The  KC-X  design  was  expected  to  draw  heavily  on  a  preexisting  commer¬ 
cial  aircraft  design.  This  presents  a  challenge  to  defining  acquisition  requirements  if  the  Air 
Force  wants  to  maintain  competition  and  avoid  choosing  the  winner  in  the  statement  of  its 
requirements.  The  more  precisely  the  Air  Force  stated  its  requirements  in  such  a  setting,  the 
easier  it  would  be  for  potential  offerors  to  compare  available  designs  against  the  requirements 
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and  determine  which  would  likely  win.  Unless  an  RFP  left  enough  room  for  at  least  two  offer¬ 
ors  to  make  serious  proposals,  it  would  not  be  rational  for  anyone  but  the  apparent  front  runner 
to  enter  the  competition. 

The  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  is  likely  to  induce  a  situation 
like  this  to  arise  more  often  in  major  system  developments.2  WSARA  requires  that  a  number 
of  actions,  such  as  competitive  prototyping  and  a  preliminary  design  review,  occur  before  the 
Air  Force  selects  a  source  to  complete  development  of  a  new  system  through  engineering  and 
manufacturing  development  (EMD).  Such  actions  are  designed  to  mature  systems  offered  for 
EMD.  Such  maturation  in  all  likelihood  will  increase  public  understanding  of  the  systems 
being  offered,  complicating  the  development  of  requirements  that  can  sustain  competition 
during  a  source  selection  in  the  same  way  that  public  knowledge  of  capabilities  in  the  CSAR-X 
and  KC-X  source  selection  complicated  those. 

One  might  argue  that,  if  the  Air  Force  understands  its  requirements  clearly  and  those 
requirements  point  to  one  design,  no  competition  is  necessary.  But  political  pressures  in  both 
of  these  source  selections  made  effective  competition  a  goal  in  its  own  right.  The  Air  Force 
was  under  heavy  pressure  to  ensure  that  competition  occurred.  In  addition,  even  if  the  Air 
Force  knows  which  proposal  it  expects  to  prefer,  competition  can  help  impose  discipline  on 
the  offeror  behind  this  proposal  to  give  the  Air  Force  the  proposed  system  on  reasonable,  cost- 
effective  terms.  Throughout  the  CSAR-X  and  KC-X  source  selections,  senior  Air  Force  officials 
made  it  clear  that  they  did  not  have  a  preferred  outcome.  But  even  if  they  had,  competition 
would  have  been  desirable  for  the  reasons  stated  above.  Knowing  this,  the  Air  Force  had  to 
frame  its  requirements  with  exceptional  subtlety  to  induce  enough  offerors  to  participate  in 
each  of  these  source  selections.  This  subtlety  became  problematic  in  the  KC-X  acquisition.  The 
Air  Force  should  not  expect  this  problem  to  be  as  serious  in  circumstances  where  the  attributes 
of  potential  proposed  designs  were  not  so  well  documented  in  the  public  record. 

The  high  stakes  associated  with  the  CSAR-X  and  KC-X  acquisitions  also  created  excep¬ 
tional  circumstances  in  both  cases.  First,  each  acquisition  had  a  high  dollar  value.  In  the  KC-X, 
this  value  was  magnified  by  the  expectation  of  two  follow-on  acquisitions  of  approximately 
equal  value;  success  in  the  first  would  likely  support  further  success  in  the  follow-ons.  Second, 
the  winner  of  each  competition  was  likely  to  emerge  as  the  global  front-runner  for  the  technol- 
ogy  in  question,  gaining  an  advantage  not  only  in  the  U.S.  market  but  in  markets  for  other 
nations  seeking  the  same  capability.  Technology  maturation  and  learning  in  production  would 
give  the  winner  an  advantage  in  future  competitions  that  any  challenger  would  have  difficulty 
overcoming.  In  the  extreme,  a  failure  in  either  source  selection  could  foreclose  a  loser’s  future 
business  opportunities  in  a  large  share  of  its  potential  markets. 

In  the  face  of  these  considerations,  the  chief  executive  officer  (CEO)  of  any  losing  offeror 
would  have  a  fiduciary  obligation  to  pursue  a  protest,  even  if  the  chances  of  success  were  small. 
The  stakes  associated  with  each  of  these  source  selections  were  so  high  that  even  a  costly  cam¬ 
paign  to  protest  the  decision  could  be  not  only  worthwhile  from  the  loser’s  point  of  view  but 
absolutely  necessary  to  satisfy  stockholders.  These  stakes  make  it  easy  to  appreciate  the  inten¬ 
sity  and  sophistication  of  the  protest  efforts  launched  in  both  these  source  selections.  It  is  hard 
to  imagine  that  any  traditional  government  jawboning  could  have  dissuaded  the  losers  from 
pursuing  these  efforts. 


2  U.S.  Congress,  Weapon  Systems  Acquisition  Reform  Act  of  2009,  P.L.  111-124,  Washington,  D.C.,  May  2009a. 
Appendix  A  provides  a  short  summary  of  WSARA  and  a  discussion  of  its  implications  for  the  issues  raised  in  the  text. 
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Finally,  each  of  these  source  selections  included  a  significant  role  for  foreign  offerors. 
The  Air  Force  was  clear  throughout  both  that  it  had  no  position  on  whether  an  American  or 
a  foreign  offeror  prevailed.3  But  the  potential  of  a  foreign  winner  raised  congressional  interest 
in  both  competitions  and  created  a  heated  political  environment  that  heightened  the  stakes 
of  these  competitions  in  the  public  policy  forum  still  higher.  We  found  no  evidence  that  this 
congressional  interest  affected  decisionmaking  within  the  Air  Force  or  GAO  in  either  source 
selection.  But  individual  decisionmakers  do  not  deliberate  in  a  vacuum,  and  those  responsible 
for  the  source  selections  themselves  and  GAO’s  review  of  them  could  not  avoid  being  affected 
by  arguments  raised  in  the  external  debate.  This  probably  made  each  of  these  source  selections 
more  complicated  and  more  difficult  to  review  objectively  than  a  typical  acquisition  of  similar 
size,  but  lacking  any  foreign  participation,  would  be. 

In  sum,  there  are  many  reasons  to  believe  the  CSAR-X  and  KC-X  acquisitions  were  more 
prone  to  protest  than  typical,  large  Air  Force  source  selections.  That  said,  the  Air  Force  should 
not  be  complacent  about  the  potential  for  similar  problems  in  the  future. 


3  The  RFPs  for  both  procurements  included  mandatory  Buy  American  Act  (BAA)  provisions.  These  provisions  exempt 
contractors  in  “qualifying  countries”  from  the  BAA,  treating  them  the  same  as  U.S.  companies  for  contract  award  purposes. 
The  political  debate  revealed  no  general  awareness  of  this  treatment  or  its  role  in  America’s  broader  program  to  promote  the 
exports  of  U.S.  defense  firms.  U.S.  firms  are  treated  (reciprocally)  as  domestic  contractors  in  procurements  that  occur  in 
qualifying  countries. 
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USAF  Will  Continue  to  Face  Challenging 
Protests  in  High-Value  Acquisitions 

•Sophisticated  protesters  are  learning  how  to  achieve 
sustainments. 

•  Knowledgeable  third-party  bar  is  building  its  skills. 

•  Sophisticated  protests  are  costly,  but 

-Worthwhile  in  high-value  acquisitions,  and 

-  Higher  cost  may  place  USAF  at  disadvantage  if  it 
does  not  expand  its  capabilities  to  defend  itself. 

•  Rate  of  sophisticated  protests  could  grow. 

If  sophisticated  protesters  understand  GAO’s 
rules  better  than  USAF  does,  they  will  win 

*AND 
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USAF  Will  Continue  to  Face  Challenging  Protests  in  High-Value  Acquisitions 

GAO  quickly  dismisses  the  majority  of  protests,  because  the  protesters  make  obvious  errors. 
These  “naive”  protests  impose  costs  on  the  Air  Force,  but  the  costs  are  small.  And  much  of  the 
improvement  in  the  Air  Force’s  protest  experience  during  the  1990s  came  from  a  reduction  in 
these  naive  protests  and  in  the  resulting  need  for  the  Air  Force  to  deal  with  them  until  GAO 
dismissed  them. 

The  protests  associated  with  the  CSAR-X  and  KC-X  acquisitions  were  anything  but  naive. 
But,  as  noted  above,  they  were  exceptional  in  many  ways.  The  main  thing  that  the  Air  Force 
can  learn  from  the  pattern  of  protests  that  we  have  described  is  that  the  sophisticated  nature 
of  the  CSAR-X  and  KC-X  protests  is  likely  the  product  of  circumstances  that  the  Air  Force 
will  face  again,  even  without  the  complications  of  subtle  requirements  and  foreign  sources  that 
make  these  two  acquisitions  especially  exceptional. 

Sophisticated  protest  campaigns  are  costly.  As  a  result,  we  should  expect  comparable  pro¬ 
tests  only  in  circumstances  where  the  expected  benefits  of  a  protest  campaign  exceed  its  cost 
to  a  losing  offeror.  As  argued  above,  that  is  more  likely  to  occur  when  the  net  revenue  associ¬ 
ated  with  winning  a  source  selection  is  large  enough.  And  it  is  more  likely  when  the  stakes  of 
losing — for  example,  potentially  being  shut  out  of  important  business  in  the  future — are  large 
enough. 

That  said,  evidence  from  the  two  acquisitions  we  explored  in  depth  suggests  that  sophis¬ 
ticated  protesters  are  learning  how  to  achieve  GAO  sustainments  in  merit  reviews.  Certainly, 
attorneys  associated  with  the  CSAR-X  protest  benefited  significantly  from  that  experience 
when  they  later  participated  in  the  KC-X  protest.  And  the  private  bar  is  probably  becoming 
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increasing  capable  of  supporting  or  even  designing  such  campaigns.  As  the  ability  of  losing 
offerors  to  protest  successfully  improves,  the  balance  between  future  net  revenues  at  stake  and 
campaign  costs  shifts,  justifying  sophisticated  protests  in  circumstances  when  smaller  future 
net  revenues  are  in  play.  Further  learning  could  bring  the  level  of  stakes  required  to  justify 
sophisticated  protest  campaigns  down  even  lower.  The  more  this  occurs,  the  more  likely  the 
Air  Force  will  face  sophisticated  protests  in  the  future. 

It  is  likely  that  one  thing  sophisticated  protesters  are  learning  is  that  it  is  hard  for  the  Air 
Force  to  surge  its  resources  to  counter  sophisticated  protests.  The  Air  Force  has  a  large  staff  of 
attorneys  who  can  be  deployed  as  needed  and  is  increasing  its  staff  to  enhance  legal  support 
to  source  selections.  Program  offices  can,  in  principle,  call  on  core  resources  in  a  similar  way 
but  in  the  protests  we  examined  in  detail,  they  had  difficulty  doing  this.  The  workload  on  one 
of  the  programs  was  so  high  that  it  had  great  difficulty  moving  beyond  the  protests  discussed 
here.  A  lesson  that  protesters  might  take  away  is  that  an  aggressive  protest  can  stress  Air  Force 
capabilities.  This  could  potentially  encourage  protests  explicitly  designed  to  overwhelm  avail¬ 
able  Air  Force  resources  in  an  effort  to  force  errors  that  protesters  can  exploit  with  resources 
that  they  can  expand  rapidly  when  errors  become  apparent.  It  is  possible  that  the  discovery  of 
this  Air  Force  weakness,  in  itself,  may  help  explain  an  increased  willingness  to  pursue  sophis¬ 
ticated  protests. 

GAO  normally  recommends  that  the  Air  Force  reimburse  a  successful  protester  for 
“allowable  costs”  of  protesting.4  These  can  cover  a  large  portion  of  all  legal  costs,  even  if  GAO 
sustains  only  one  of  the  many  grounds  for  protest  that  attorneys  have  developed  for  the  pro¬ 
tester.  This  GAO  policy  tends  to  push  any  cost-benefit  calculus  toward  supporting  a  protest 
if  an  offeror  believes  that  it  has  a  reasonable  chance  of  prevailing  on  at  least  one  ground  for 
protest.  In  particular,  if  a  sophisticated  protester  believes  that  it  has  found  ways  to  increase  the 
probability  of  having  a  protest  sustained  in  the  presence  of  any  given  set  of  legal  facts,  GAO’s 
reimbursement  policy  tends  to  push  any  cost-benefit  calculus  still  further  toward  supporting 
a  protest. 

In  the  face  of  this  threat  from  sophisticated  protesters,  the  Air  Force  has  a  simple  and 
direct  defense — learn  enough  about  how  these  campaigns  work  so  that  it  can  limit  their  suc¬ 
cess,  either  by  foreclosing  opportunities  for  protest  during  the  design  and  execution  of  a  source 
selection  or  by  being  prepared  for  the  protests  when  they  come.  A  failure  to  build  such  a 
defense  will  likely  encourage  still  more  sophisticated  protests  and  more  sustainments  against 
the  Air  Force’s  interests. 


4  If  an  agency  takes  corrective  action  within  30  days  of  the  date  of  filing  of  the  protest,  the  protester  is  not  entitled  to 
recover  attorneys  fees  or  protest  costs. 
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A  Bid  Protest  Is  an  Adversarial 
Proceeding  with  Finely  Tuned  Rules 

*  Goals  of  a  GAO  proceeding  differ  from  USAF  goals: 

-  USAF  acquisition  seeks  to  balance  priorities  of 
regulations  and  the  warfighter. 

-  GAO  considers  only  federal  acquisition  law. 

-  Result:  Many  in  USAF  question  GAO’s  motives. 

*  USAF  would  benefit  from  a  better  understanding  of 
GAO’s  impersonal,  judicial-like  approach. 

-Attorneys  for  protesters  use  such  an 
understanding  to  scour  USAF  documents  for 
errors. 

-  USAF  could  include  attorneys  in  a  red  team  to 
anticipate  and  preempt  this  effort. 

ID 
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A  Bid  Protest  Is  an  Adversarial  Proceeding  with  Finely  Tuned  Rules 

In  our  discussions  with  Air  Force  personnel,  most  appeared  to  be  surprised,  confused,  or 
even  offended  by  the  difference  in  the  priorities  of  the  Air  Force  and  GAO  in  a  bid  protest. 
Air  Force  acquisition  personnel  understand  the  importance  of  the  acquisition  regulations  that 
GAO  enforces,  but  they  also  appreciate  the  importance  of  rendering  decisions  that  ultimately 
serve  the  warfighter.  When  ambiguities  arise,  they  may  not  notice  them,  because  the  attention 
they  give  to  the  warfighter  helps  them  clarify  the  ambiguities  in  ways  that  seem  to  embody 
common  sense  for  most  personnel  with  a  similar  perspective.  For  example,  if  disagreements 
arise  about  how  to  interpret  the  language  in  an  RFP  during  the  evaluation  of  proposals,  Air 
Force  personnel  often  find  it  natural  and  reasonable  to  resolve  such  disagreements  in  ways  that 
favor  the  clear  priorities  of  the  warfighter.  Military  practice  emphasizes  that  few  plans  survive 
their  first  contact  with  the  enemy;  inherent  uncertainty  demands  adaptability,  informed  by 
constant  awareness  of  the  higher  priorities  of  the  military  mission. 

That  is  not  how  GAO  views  the  world.  GAO  draws  its  authority  directly  from  congres¬ 
sional  legislation  seeking  to  highlight  the  inherent  importance  of  federal  acquisition  policy. 
When  the  Congress  considered  the  best  ways  to  do  this,  one  option  was  to  invest  the  responsi¬ 
bility  for  bid  protests  in  the  inspectors  general  of  the  individual  military  services  where  acqui¬ 
sitions  occurred.  Congress  explicitly  rejected  this  option  and  instead  gave  the  responsibility  to 
an  independent  third  party — GAO.  It  did  this  to  ensure  that  the  principles  of  the  acquisition 
regulations  were  given  adequate  attention,  quite  separate  from  the  military  missions  of  the  ser¬ 
vices  in  which  acquisition  occurred.  That  is,  GAO  draws  its  authority  to  review  bid  protests 
precisely  from  a  congressional  judgment  that  GAO  should  not  rely  on  military  judgment  to 
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resolve  uncertainty.  Rather,  as  an  independent  agency,  GAO  should  focus  on  the  language  in 
any  particular  source  selection  to  speak  for  itself  and  to  interpret  this  language  as  a  reasonable 
person  would  without  falling  back  on  military  priorities  when  the  language  itself  is  unclear. 

That  is  not  to  say  that  GAO  will  not  defer  to  military  judgment.  In  fact,  in  the  majority 
of  merit  reviews,  GAO  shows  great  deference  to  military  judgment  where  the  source  selection 
documents  clearly  define  the  role  for  such  judgment  and  the  documents  underlying  evaluation 
and  the  final  choice  of  a  winner  clearly  explain  the  application  of  that  judgment. 

Rather,  when  the  language  in  these  documents  speaks  clearly  for  itself,  GAO  accepts  this 
language  on  its  own  terms.  When  the  language  does  not  speak  clearly,  GAO  does  not  resort  to 
the  military  reasoning  that  might  have  resolved  any  uncertainty  in  a  purely  Air  Force  setting. 
In  fact,  GAO  views  its  congressional  mandate  as  requiring  judgment  quite  separate  from  such 
military  judgment  when  judgment  is  required  to  resolve  uncertainty  in  a  source  selection. 

Most  of  the  Air  Force  personnel  we  spoke  to  greet  such  a  perspective  with  marked  cul¬ 
tural  dissonance.  To  them,  it  seems  technical,  counterintuitive,  and  not  supportive  of  the  mili¬ 
tary  mission  that  most  Air  Force  personnel  take  for  granted.  Just  beneath  the  surface,  one 
can  feel  a  certain  suspicion  that  the  Air  Force  and  GAO  are  not  on  the  same  team.  Viewed 
appropriately,  that  suspicion  is  correct.  GAO  and  the  Air  Force  have  separate  and  distinct  mis¬ 
sions  that  naturally  come  into  conflict  in  bid  protests.  To  be  as  effective  as  possible  in  bid  pro¬ 
tests,  the  Air  Force  personnel  in  question  need  to  understand  this  difference  in  perspective,  to 
understand  the  rights  that  GAO  gives  the  Air  Force  in  its  bid  protest  process,  and  to  take  full 
advantage  of  those  rights. 

GAO’s  bid  protest  process  is  purely  administrative,  but  GAO  has  chosen  to  run  it  in  a 
quasi-judicial  way.  GAO  tries  to  focus  on  procedural  issues  without  involving  itself  in  the  sub¬ 
stance  of  the  issues  raised  in  a  source  selection.  To  do  that,  it  views  itself  as  an  impersonal  arbi¬ 
ter,  simply  applying  the  statutory  guidance  that  defines  its  mission  to  the  specific  facts  of  indi¬ 
vidual  bid  protests.  It  uses  standard  administrative  methods  to  promulgate  rules  of  procedure 
that  are  consistent  with  its  enabling  legislation.  It  does  not  use  such  methods  to  promulgate 
specific  rules  on  what  government  agencies  can  or  cannot  do  in  specific  circumstances  during 
a  source  selection  to  remain  in  compliance  with  this  legislation.  It  is  simply  the  creature  of  the 
congressional  rules  that  it  applies. 

That  said,  the  history  of  GAO’s  bid  protest  decisions  yields  a  record  of  how  GAO  inter¬ 
prets  the  congressional  rules  that  it  enforces.  When  it  explains  its  decision  in  one  bid  protest, 
it  relies  heavily  on  decisions  it  has  made  in  similar  cases  in  the  past  to  explain  the  reasoning 
underlying  the  new  decision.  The  record  of  GAO  decisions  closely  resembles  that  generated  by 
judicial  decisions  in  courts.  The  record  itself  provides  a  public  statement  of  how  GAO  officials 
think.  GAO  has  been  careful  not  to  offer  any  formal  statements  of  “how  GAO  thinks”  outside 
the  context  of  specific  decisions  that  apply  congressional  rules  to  new  sets  of  facts.  That  allows 
GAO  to  avoid  any  responsibility  as  a  formal  rule  maker,  with  all  its  attendant  administrative 
responsibilities.  But  as  a  practical  matter,  GAO’s  decisions  explain  how  it  approaches  indi¬ 
vidual  bid  protests  and,  in  doing  so,  reveal  a  set  of  implicit  rules  that  the  Air  Force  can  use  to 
understand  what  to  expect  in  any  new  bid  protest. 

As  a  practical  matter,  this  approach  tends  to  refine  GAO’s  perspective  over  time.  We 
should  expect  merit  reviews  only  when  some  degree  of  uncertainty  prevails  about  how  GAO 
will  rule.  If  everyone  agrees  on  how  GAO  would  treat  any  new  set  of  facts,  no  one  benefits 
from  going  forward  with  a  merit  review.  In  this  setting,  as  GAO  faces  more  and  more  indi¬ 
vidual  sets  of  facts  in  bid  protests  over  time,  each  new  set  of  facts  gives  GAO  a  potential  oppor- 
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tunity  to  clarify  its  interpretation  of  the  congressional  rules  it  applies  in  a  way  that  helps  resolve 
the  uncertainty  that  led  to  the  merit  review.  Over  time,  this  approach  necessarily  requires  that 
GAO  clarify  distinctions  between  situations  with  different  sets  of  facts  and  thereby  surfaces 
more  and  more  precise  GAO  judgments  about  the  practical  meaning  of  congressional  rules.  As 
this  occurs,  it  is  likely  that  bid  protests  today  turn  on  finer  distinctions — are  more  technical 
in  character — than  bid  protests  in  the  past.  So  it  would  be  reasonable  to  expect  that  GAO’s 
decisions  tend  to  become  increasingly  technical  in  character  over  time.  To  the  degree  that  this 
occurs,  the  Air  Force  should  not  conclude  that  GAO  review  officials  are  consciously  becoming 
less  deferential  or  more  picky.  Rather,  it  is  a  natural  consequence  of  how  GAO  makes  decisions. 
It  will  continue.  As  a  result,  it  behooves  the  Air  Force  to  accept  this  trend  and  accommodate 
itself.  The  better  the  Air  Force  can  anticipate  how  GAO  will  treat  a  new  set  of  facts,  the  better 
able  the  Air  Force  will  be  to  avoid  protests  in  the  first  place  and  respond  to  them  when  they 
occur. 

GAO  seeks  to  focus  on  procedure  to  avoid  second-guessing  the  Air  Force’s  substantive 
decisionmaking.  This  can  be  difficult  when  GAO  seeks  to  assess  how  reasonable  the  Air  Force 
has  been  in  evaluating  a  proposal.  Three  of  the  ten  grounds  for  sustainment  in  the  CSAR-X 
and  KC-X  source  selections  raised  questions  about  whether  the  Air  Force’s  approach  to  cost 
estimation  was  “reasonable.”  In  its  discussion  of  these  sustainments,  GAO  made  strong  state¬ 
ments  about  how  the  Air  Force  should  think  about  cost  that,  in  the  KC-X  sustainments,  ulti¬ 
mately  directed  the  Air  Force  how  to  estimate  cost.  GAO’s  guidance  appears  to  carry  GAO 
deep  into  the  substance  of  how  the  Air  Force  makes  decisions  about  source  selections;  we  dis¬ 
cuss  the  issues  raised  in  greater  depth  below. 

GAO’s  review  process  also  necessarily  extends  beyond  procedural  issues  when  it  deter¬ 
mines  whether  an  Air  Force  error  has  “prejudiced”  any  offeror  in  a  source  selection.  This  deter¬ 
mination  does  not  directly  shape  how  the  Air  Force  should  correct  any  specific  error.  But  it 
directly  affects  whether  GAO  recommends  any  correction  at  all.  “Prejudice”  exists  when  an 
error  is  serious  enough  to  change  the  outcome  of  the  source  selection.  If  prejudice  is  not  pres¬ 
ent,  GAO  will  not  sustain  a  protest.  How  does  GAO  decide  when  prejudice  exists?  It  has  no 
formal  rules  or  guidelines  for  its  review  officials.  In  the  context  of  how  the  Air  Force  has  applied 
the  terms  of  an  RFP,  to  reach  substantive  judgments  on  who  should  win  and  why,  GAO  offi¬ 
cials  must  judge  whether  any  particular  error  would  affect  the  balance  of  substantive  Air  Force 
judgments  enough  to  affect  the  final  decision.  Even  if  GAO  does  not  question  the  substance  of 
most  Air  Force  judgments,  it  must  assess  how  the  substantive  implications  of  an  error  fit  in  this 
broader  substantive  context — in  effect,  what  substantive  change  the  Air  Force  would  make  in 
its  own  judgments  if  it  corrected  the  error.  That  is,  even  where  GAO  is  concerned  solely  about 
procedural  errors,  GAO  decisions  to  sustain  a  protest  based  on  a  procedural  error  necessarily 
depend  directly  on  GAO  judgments  about  Air  Force  decisions  on  substantive  issues.  Those 
judgments  are  likely  to  conclude  that  any  error  is  prejudicial  in  a  close  procurement. 

At  the  end  of  the  day,  the  Air  Force  should  not  be  dismayed  that  GAO  is  not  friendlier 
or  fails  to  defer  regularly  to  Air  Force  military  judgment.  GAO  clearly  views  the  Air  Force  as 
one  participant  in  an  adversarial  proceeding  very  much  like  that  in  a  normal  courtroom.  The 
main  practical  difference  is  that  the  Air  Force  lacks  some  of  the  basic  rights  that  a  party  would 
have  in  a  courtroom.  That  is  unfortunate  and  perhaps  even  unfair.  But  because  any  attempt  to 
change  that  would  likely  increase  the  administrative  burden  imposed  by  the  bid  protest  pro¬ 
cess,  GAO  is  unlikely  to  change  the  balance  of  power  in  the  current  process.  And  Congress 
would  probably  not  support  Air  Force  efforts  to  make  such  a  change.  Given  the  Air  Force’s 
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generally  improving  overall  experience  with  bid  protests  under  the  current  system,  even  the  Air 
Force  itself  might  regret  asking  for  basic  changes  that  would  necessarily  complicate  bid  protests 
for  everyone. 

Given  the  basically  legal  nature  of  the  process  as  it  exists  today  and  is  likely  to  continue  to 
be  in  the  future,  it  is  natural  to  imagine  an  expanded  role  for  attorneys  in  the  process.  Attor¬ 
neys  were  already  heavily  involved  in  two  grounds  for  sustained  protests  in  the  CSAR-X  and 
KC-X  source  selections.  The  Air  Force’s  effort  to  respond  to  the  first  sustained  protest  in  the 
CSAR-X  case  by  limiting  the  ability  of  offerors  to  adjust  their  proposals  was  heavily  vetted 
by  attorneys  in  the  Air  Force  and  OSD.  Similarly,  when  the  KC-X  program  office  decided  to 
downgrade  its  evaluation  of  Boeing  on  one  subfactor  without  discussions,  it  heavily  vetted  this 
decision  with  Air  Force  attorneys,  who  endorsed  the  decision  on  narrow  technical  grounds. 
The  role  of  the  attorney  is  to  advise  the  client.  The  client  must  make  the  decision  as  to  how  to 
proceed.  The  challenge  for  Air  Force  attorneys  and  decisionmakers  today  is  to  learn  appropriate 
lessons  from  these  decisions  so  that  GAO  is  less  likely  to  sustain  similar  protests  in  the  future. 

The  Air  Force  is  already  expanding  the  numbers  of  attorneys  it  has  available  to  support 
source  selections  in  the  future.  This  could  make  it  easier  for  Air  Force  attorneys,  and  poten¬ 
tially  for  third-party  attorneys  working  under  the  direction  of  Air  Force  attorneys,  to  pursue 
new  opportunities.  Although  GAO  initially  designed  the  process  to  allow  protesters  to  use  it 
without  legal  representation,  sophisticated  protesters  are  now  using  highly  skilled  attorneys 
working  hard  to  take  full  advantage  of  its  idiosyncrasies.  For  example,  the  waves  of  grounds  for 
protest  that  both  the  CSAR-X  and  KX-C  programs  experienced  show  clear  marks  of  an  orches¬ 
trated  process  to  generate  grounds  for  protest  designed  in  part  to  release  significant  numbers 
of  government  documents  that  would  likely  provide  material  for  still  more  grounds  for  protest. 
The  grounds  for  protest  offered  show  the  marks  of  subtle  legal  thinking  designed  to  appeal  to 
the  GAO  attorneys  who  will  make  final  decisions  on  sustainments  and  rejections. 

A  government  red  team,  including  attorneys,  could  proactively  scour  decisions  and  docu¬ 
ments  generated  during  Air  Force  source  selections  seeking  to  anticipate  how  a  protester’s 
attorneys  and  advisors  would  do  the  same  thing  after  a  protest  starts.  Such  a  team  might  also 
include  engineers,  logisticians,  contracting  specialists,  cost  analysts,  and  so  on;  our  discus¬ 
sion  here  focuses  on  the  expanded  role  that  attorneys  might  play.5  This  review  would  not  be  a 
standard  review  designed  to  ensure  orderly  program  execution.  Rather,  it  would  deliberately 
seek  weak  points,  expecting  that  protesters  will  use  any  weak  points  they  discover  to  leverage 
their  resources  to  greatest  effect.  That  is,  it  would  anticipate  that,  in  source  selections  likely  to 
involve  sophisticated  protesters,  the  end  game  will  occur  in  an  adversary  process  in  which  the 
Air  Force  will  be  asked  to  defend  all  of  its  actions  through  the  course  of  the  source  selection. 

This  approach  is  likely  to  be  most  effective  if  the  Air  Force  can  shield  the  deliberations  of 
red  teams  from  the  official  record  of  the  source  selection  provided  to  a  protester.6  Red  teams 
will  always  have  limited  resources  and  will  have  to  make  decisions  about  where  to  focus  their 
attention.  Discussions  will  inevitably  arise  about  whether  the  structure  of  an  RFP  is  “good 
enough,”  and  decisions  will  have  to  be  made  even  in  the  absence  of  consensus.  If  a  protester 


5  That  said,  many  of  the  points  raised  here  about  attorneys  apply  to  other  potential  participants  on  a  red  team.  The  Air 
Force  currently  uses  review  teams  with  such  personnel.  The  red  team  we  describe  would  not  replace  these  teams.  It  could 
act  as  an  integral  part  of  existing  review  teams. 

6  Today,  the  Air  Force  releases  multifunctional  independent  review  team  (MIRT)  and  peer  review  findings  to  protesters. 
The  program  office  must  carefully  document  how  it  reacts  to  recommendations  from  these  reviews. 
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can  look  at  where  a  red  team  focused  its  attention  or  review  the  statements  of  dissenters  in  this 
process,  such  information  would  become  the  natural  basis  for  a  check  list  that  a  protester  could 
use  to  identify  weak  points. 

Today,  when  a  protest  occurs,  the  Air  Force  benefits  from  the  attorneys  that  the  win¬ 
ning  firm  can  bring  to  bear  to  help  defend  Air  Force  decisions  made  earlier.  A  government  red 
team  would  have  to  be  constituted  and  active  long  before  the  government  revealed  winners 
and  losers.  Initial  hard  reviews  would  occur  before  the  Air  Force  released  a  final  RFP.  Equally 
demanding  reviews  would  occur  during  the  evaluation  phase — perhaps  just  before  the  call  for 
FPRs — and  before  approval  of  the  final  decision.  The  Air  Force  would  have  to  rely  on  its  own 
resources  to  generate  such  a  capability. 

Adding  attorneys  and  steps  in  the  source  selection  process  in  which  attorneys  can  review 
Air  Force  actions  would  almost  certainly  raise  the  administrative  costs  of  the  source  selection 
and  add  calendar  time  to  its  schedule.  Additional  attorneys  cost  money  to  train  and  sustain; 
this  expense  will  be  easiest  to  justify  if  these  attorneys  can  reduce  other  costs  still  more.  Simi¬ 
larly,  because  the  tasks  that  attorneys  perform  will  almost  surely  be  on  the  critical  path  of  any 
source  selection,  it  will  be  easiest  for  the  Air  Force  to  justify  their  presence  there  if  they  can 
shorten  the  calendar  time  associated  with  other  steps  on  the  critical  path — for  example,  the 
calendar  time  required  to  complete  evaluation  or  deal  with  a  sustained  protest.  The  protest- 
induced  monetary  costs  and  delays  experienced  in  the  CSAR-X  and  KC-X  source  selections 
were  large  relative  to  the  added  cost  and  delay  one  might  hypothetically  associate  with  any 
legal  red  team.  But  when  asking  how  large  a  red  team  should  be  or  how  many  calendar  days 
the  Air  Force  should  give  it  for  review,  as  the  Air  Force  adds  more  and  more  person-days,  the 
incremental  benefit  of  an  additional  legal  person-day  will  ultimately  fall  so  that  it  is  lower  than 
the  cost  the  Air  Force  expects  for  such  a  person-day. 

Such  considerations  make  it  clear  that  any  red  team  activity  must  be  coordinated  with 
other  oversight  activities,  such  as  MIRTs,  to  avoid  redundancy  and  limit  the  burden  placed 
on  the  people  actually  conducting  the  source  selection.  We  conceive  the  deliberate  search  for 
weaknesses  as  different  in  character  from  the  compliance  perspective  that  tends  to  dominate 
most  external  oversight,  but  no  bright  line  separates  such  activities.  For  example,  closely  match¬ 
ing  the  information  requested  in  a  source  selection  to  the  methods  that  the  source  selection 
will  use  to  apply  this  information  in  evaluation  would  be  a  natural  activity  to  include  in  either 
approach. 

This  red-teaming  capability  need  not  be  wholly  organic.  The  Air  Force  legal  community 
has  been  reluctant  to  use  contract  attorneys  to  perform  legal  tasks  that  could  easily  be  regarded 
as  inherently  governmental.7  The  red  team  envisioned  here  involves  no  activities  that  could 


7  The  FAR  defines  the  formal  meaning  of  an  “inherently  government  activity”  in  Subpart  7.5.  The  Air  Force  General 
Counsel  has  concluded  that  legal  services  provided  directly  to  government  officials  performing  inherently  governmental 
functions,  such  as  acquisition  professionals,  are  inherently  governmental  themselves  (personal  communication  with  James 
Hughes,  Air  Force  Deputy  General  Counsel  for  Acquisition  [SAF/GCQJ).  The  Office  of  Management  and  Budget  recently 
proposed  restricting  the  type  of  activities  contractors  can  perform  to  extend  beyond  inherently  governmental  activities  in 
a  number  of  ways  (Office  of  Management  and  Budget,  Office  of  Federal  Procurement  Policy,  “Work  Reserved  for  Perfor¬ 
mance  by  Federal  Government  Employees,”  Notice  of  Proposed  Policy  Letter  in  Federal  Register,  Vol.  75,  No.  61,  March 
31,  2010,  pp.  16188-16197):  “Functions  closely  associated  with  the  performance  of  inherently  governmental  functions,” 
for  example,  could  become  off-limits  to  contractor  provision  if  contractor  provision  “impingefd]  on  federal  officials’  per¬ 
formance  of  an  inherently  governmental  function.”  Such  activities  could  include  “technical  advisors  to  a  source  selection” 
and  “drafting  of  legal  advice.”  The  proposed  guidance  would  also  define  “critical  functions”  that  are  “core  to  the  agency’s 
mission  and  Operations,  .  .  .  whose  importance  to  the  agency’s  mission  and  operation  requires  that  at  least  a  portion  of  the 
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be  construed  as  inherently  governmental.  It  is  strictly  an  advisory  activity  with  no  authority, 
for  example,  to  exercise  discretion  in  applying  federal  government  authority.  In  fact,  external 
attorneys  might  be  more  effective  in  this  role,  because  they  might  show  less  deference  to  the 
government  personnel  working  within  a  source  selection  and,  if  drawn  from  the  same  private 
bar  that  advises  sophisticated  protesters,  might  be  exceptionally  knowledgeable  about  how 
these  protesters  operate.* * * * * * 8 

Use  of  third-party  attorneys  is  not  a  perfect  solution.  They  are  expensive  relative  to  gov¬ 
ernment  attorneys  with  comparable  experience.  Even  junior  attorneys  carry  heavy  indirect 
costs,  and  the  real  benefit  from  external  attorneys  would  come  from  attorneys  with  appropriate 
experience.  Conflict  of  interest  issues  would  have  to  be  carefully  addressed  with  experienced 
attorneys  to  separate  those  supporting  the  government  from  those  supporting  protesters.  The 
bars  of  all  states  and  the  federal  bar  all  have  clear  and  strict  ethics  rules  about  conflicts  of  rep¬ 
resenting  clients.9  But  the  application  of  these  rules  to  our  setting  might  limit  the  government’s 
access  to  the  best  attorneys  available  on  the  market.  These  problems  will  presumably  become 
less  compelling  as  the  Air  Force  grows  the  number  of  in-house  attorneys  who  support  source 
selections. 

The  government  should  also  be  cognizant  that,  despite  any  conflict  of  interest  agreement, 
work  for  the  government  today  could  teach  an  attorney  or  law  firm  how  to  support  a  protester 
in  the  future.  The  Air  Force  saw  this  occur  at  close  quarters  in  the  CSAR-X  and  KC-X  protests 
when  attorneys  supporting  Boeing  in  the  CSAR-X  case  and  supporting  the  Air  Force  in  the 
CSAR-X  protests  used  what  they  learned  about  the  Air  Force  in  that  source  selection  to  mount 
an  effective  sophisticated  protest  for  Boeing  in  the  KC-X  source  selection.  That  is,  private- 
sector  attorneys  are  already  working  very  close  to  the  government  in  bid  protests  and  learning 
about  how  government  processes  work.  They  do  not  have  to  work  directly  for  the  government 
to  do  that.  Nonetheless,  the  Air  Force  might  want  to  explore  appropriate  terms  of  engagement 
before  bringing  private-sector  attorneys  even  closer  to  the  Air  Force  source  selection  process. 


function  must  be  reserved  to  federal  employees  in  order  to  ensure  the  agency  has  sufficient  internal  capability  to  effectively 

perform  and  maintain  control  of  its  mission  and  operations.”  Only  government  personnel  could  perform  such  functions. 

These  restrictions  have  not  been  finalized.  In  their  current  draft  form,  they  appear  more  likely  to  restrict  use  of  third-party 

attorneys  on  red  teams  because  such  teams  might  be  considered  too  “closely  associated  with  the  performance  of  inherently 

governmental  functions”  than  because  they  were  themselves  critical  functions  (Office  of  Management  and  Budget,  “Notice 

of  Proposed  Policy  Letter,”  2010. 

8  By  similar  reasoning,  it  would  be  undesirable  to  use  military  retirees  or  former  Air  Force  employees  on  such  red  teams, 
because  they  would  likely  be  too  close  in  their  thinking  to  the  SPO’s  approach  and  would  likely  share  its  biases  and  blind 
spots.  The  goal  is  to  find  red  team  participants  familiar  with  the  practices  and  culture  of  aggressive  protesters,  not  the  prac¬ 
tices  and  culture  of  the  Air  Force.  We  thank  Elliot  Axelband  for  this  insight. 

9  We  thank  Bruce  Field  for  this  insight  and  others  about  the  challenges  of  using  third-party  attorneys  in  this  setting. 
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Plans  to  Clarify  Trade-Off  Process  in  Future 
USAF  Source  Selections  Appear  Useful 

•Trade-off  process  has  pros  and  cons 

-  Pro:  Provides  framework  to  increase  performance 
in  discussions. 

-  Con:  Can  create  ambiguity  that  allows  GAO  to  say 
what  is  “reasonable.” 

•  New  draft  RFP  for  KC-X  strikes  useful  new  balance. 

-  Greater  reliance  on  pass/fail  evaluation. 

-  Emphasis  on  discriminators  in  trade  space. 

-  Clearly  stated  priorities  among  requirements. 

*AND 
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Plans  to  Clarify  Trade-Off  Process  in  Future  USAF  Source  Selections  Appear 
Useful 

Source  selections  that  trade  off  price  and  performance  have  improved  the  Air  Force’s  ability 
to  induce  offerors  to  write  proposals  that  are  responsive  to  Air  Force  priorities.10  Such  source 
selections  identify  a  set  of  requirements  and  set  a  “threshold”  and  “objective”  level  for  each 
requirement.* 11  A  proposal  must  achieve  the  threshold  for  every  requirement  listed  in  the  RFP. 
The  Air  Force  adds  incremental  credit  as  a  proposal  offers  more  than  the  threshold  level  of  a 
requirement,  up  to  the  objective  level.  The  proposal  earns  nothing  more  if  it  offers  more  than 
the  objective.  The  range  between  the  threshold  and  objective  for  each  requirement,  viewed 
across  all  requirements,  constitutes  the  trade  space  within  which  the  Air  Force  assesses  the  rela¬ 
tive  value  of  increases  in  offered  performance  for  various  requirements. 

To  use  this  method  of  source  selection  to  greatest  advantage,  the  Air  Force  requests  for 
proposals  must  identify  thresholds  and  objectives  clearly  and  explain  clearly  how  Air  Force 
evaluators  will  assess  the  value  of  increases  in  offered  performance  across  all  requirements  in 
the  trade  space.  When  an  RFP  has  done  this  well,  exchanges  between  the  Air  Force  and  its 
offerors  during  evaluation  create  a  natural  environment  in  which  the  Air  Force  can  explore 
options  with  offerors  and  negotiate  with  them  to  move  from  a  threshold  toward  an  objective 


10  U.S.  Air  Force,  2009b,  defines  this  approach  in  detail. 

11  If  the  objective  equals  the  threshold  for  a  requirement,  the  requirement  is  not  part  of  the  trade  space.  When  this  occurs, 
an  offeror  must  achieve  the  threshold  to  be  considered  responsive.  For  such  a  requirement,  the  Air  Force  provides  no  extra 
credit  for  any  performance  beyond  the  threshold. 
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in  the  most  cost-effective  way  possible.  Advocates  believe  that  this  approach  has  helped  the 
Air  Force  increase  what  it  gets  for  the  money  it  ultimately  pays  the  offerors  that  win  source 
selections. 

As  advantageous  as  this  approach  may  be,  it  probably  makes  it  easier  for  protesters  to 
have  protests  sustained.  First,  priorities  within  the  trade  space  can  be  inherently  complex  and 
ambiguous.  The  more  complex  and  ambiguous  they  are,  the  more  opportunities  protesters 
have  to  demonstrate  that  (1)  an  evaluation  deviates  from  the  RFP  criteria  used  in  a  source  selec¬ 
tion,  (2)  the  Air  Force  has  not  treated  offerors  equally,  or  (3)  the  Air  Force  has  used  unreason¬ 
able  methods  or  judgments  to  evaluate  criteria  in  the  trade  space. 

Second,  if  negotiation  reduces  the  differences  between  the  offers  proffered  in  the  FPRs, 
these  small  differences  potentially  open  more  opportunities  for  Air  Force  errors  to  change  the 
outcome  of  a  competition.  Given  ambiguity  about  how  GAO  decides  when  an  error  can  affect 
the  outcome  of  a  source  selection  in  practice,  this  second  point  is  not  as  compelling  as  the  first. 
And  some  observers  note  that,  if  negotiation  refines  all  proposals  to  the  point  where  Air  Force 
evaluation  documents  that  all  proposals  “exceed  specified  minimum  performance  or  capability 
requirements  in  a  way  beneficial  to  the  government,”  and  so  all  can  be  given  a  “blue”  rating, 
then  the  outcome  of  the  source  selection  will  turn  on  price.12  In  this  situation,  errors  that  do 
not  affect  the  blue  rating  for  any  subfactor  cannot  change  the  outcome  of  the  source  selection 
and  so  should  not  lead  GAO  to  sustain  a  protest  based  on  such  errors. 

Other  Air  Force  observers  caution  that,  even  if  all  proposals  exceed  minimum  require¬ 
ments  in  any  subfactor,  it  is  desirable  from  the  Air  Force’s  perspective  to  draw  finer  distinctions 
and  seek  to  discriminate  among  such  proposals  where  possible.  This  generally  involves  using 
a  list  of  considerations  to  rate  each  subfactor.  If  the  Air  Force  source  selection  team  cannot 
discriminate  among  proposals  on  the  basis  of  the  most  important  considerations  in  the  list,  it 
can  move  down  the  list  to  secondary  and,  if  necessary,  tertiary  considerations.13  If  the  source 
selection  team  takes  this  approach,  relatively  small  errors  in  evaluation  can  potentially  affect 
the  outcome  of  the  source  selection  and  so  prompt  a  sustained  protest. 

In  the  protests  sustained  by  GAO  that  we  examined,  we  saw  many  examples  of  this 
latter  approach  without  GAO  objection.  This  suggests  that  GAO  is  comfortable  with  an  Air 
Force  source  selection  designed  to  draw  fine  distinctions  among  proposals  when  more  than 
one  exceeds  minimum  requirements.  GAO  appears  to  be  concerned  not  with  how  fine  a  dis¬ 
tinction  the  Air  Force  draws  when  assessing  any  particular  subfactor  but  with  how  well  the 
Air  Force  evaluation  hews  to  the  description  of  potential  distinctions  presented  in  the  RFR  In 
this  setting,  finer  distinctions  open  the  door  for  smaller  errors  to  prompt  GAO  sustainments. 

One  way  to  avoid  such  exposure  is  to  avoid  the  use  of  such  fine  distinctions.  Experienced 
Air  Force  acquisition  officials  warned  us  that  such  an  approach  can  eviscerate  the  opportunities 
the  Air  Force  has  to  drive  improvement  through  negotiations.  Without  fine  distinctions,  the 
motivation  to  improve  performance  in  response  to  Air  Force  prompting  during  negotiations  is 
hard  to  sustain.  In  fact,  the  finer  the  distinctions  it  plans  to  use  in  evaluation,  the  harder  the 
Air  Force  can  press  its  offerors. 


12  U.S.  Department  of  Defense,  2009e,  Chart  39.  When  a  source  selection  yields  blue  ratings  for  all  subfactors  and  propos¬ 
als,  the  source  selection  team  faces  a  “sea  of  blue”  and  must  decide  how  to  discriminate  among  proposals  in  the  face  of  this 
outcome. 

13  Advocates  of  this  approach  argue  that  “blue  is  not  blue”;  it  is  feasible  and  desirable  to  draw  distinctions  between  propos¬ 
als  that  exceed  minimum  requirements  on  any  subfactor. 
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Some  observers  have  concluded  from  recent  experience  that  trade-off  source  selections 
are  fatally  flawed  and  that  the  Air  Force  should  abandon  them.  Rather,  it  should  use  a  lowest- 
price,  technically  acceptable  approach  that  assesses  each  proposal  to  ensure  that  it  is  techni¬ 
cally  acceptable  and  then,  among  all  technically  acceptable  proposals,  choose  the  one  with  the 
lowest  price.  This  approach  encourages  offerors  to  frame  proposals  that  achieve  the  minimum 
level  of  technically  acceptable  performance  so  that  they  can  keep  their  prices  as  low  as  pos¬ 
sible.  It  assumes  that  the  Air  Force  knows  exactly  what  to  ask  for  as  minimum  standards  and 
does  not  need  to  use  negotiation  to  learn  the  true  capabilities  of  offerors  before  setting  those 
standards. 

This  argument  does  not  appear  to  balance  the  benefits  of  a  trade-off  approach — which 
helps  the  Air  Force  get  better  systems  in  its  source  selections — with  its  costs — in  the  form  of 
more  frequent  demands  for  corrective  action  or  outright  GAO  sustainments.  Any  effort  to  bal¬ 
ance  these  costs  and  benefits  is  likely  to  yield  a  mixed  approach  designed  to  preserve  the  ben¬ 
efits  and  to  mitigate  any  costs  that  accompany  them.  Consider  some  examples. 

Limit  the  number  of  factors  included  in  the  trade  space.  Treat  most  requirements  as  pass  / 
fail  options  that  any  offeror  must  achieve — thresholds  with  no  objectives  above  the  thresholds. 
Then  use  pre-solicitation  market  research  to  identify  the  factors  that  the  Air  Force  cares  about 
that  are  most  likely  to  distinguish  offerors  from  one  another  and  build  the  trade  space  around 
a  small  set  of  these  “discriminators.”14 

State  priorities  among  items  in  the  trade  space  and  subfactors  precisely  enough  that  evaluators 
can  clearly  justify  their  assessments  and  the  distinctions  they  draw  between  offerors.  That  does  not 
imply  identifying  quantitative  weights  for  finely  distinguished  requirements.  Rather,  it  involves 
seeking  a  balance  between  (1)  definitions  in  the  RFP  that  are  clear  enough  to  allow  evaluations 
to  draw  meaningful  distinctions  and  justify  them  and  (2)  flexibility  that  allows  the  Air  Force 
to  exercise  appropriate,  inherently  subjective  military  judgment  when  it  makes  these  distinc¬ 
tions.  For  a  contracting  officer,  it  would  be  ideal  to  write  all  factors  on  a  white  board  and  start 
erasing  those  elements  that  are  not  discriminators  among  offerors.  In  that  vision  of  the  future, 
for  example,  the  Air  Force  would  not  have  included  the  MER  for  an  existing  legacy  system  in 
the  CSAR-X  source  selection,  because  it  was  not  a  discriminator  among  offerors.  This  is  not 
easy  to  do  and  involves  skills  that  are  not  plentiful  in  the  Air  Force  acquisition  workforce  today. 
Close  application  of  the  expertise  that  the  Air  Force  does  have  would  be  beneficial. 

To  the  extent  that  discussions  do  erase  too  much  of  the  distinction  between  final  pro¬ 
posals,  the  Air  Force  must  weigh  the  advantages  of  (1)  negotiating  to  the  point  where  the  Air 
Force  would  be  happy  with  any  of  the  offers  and  (2)  preserving  enough  difference  between 
offers  that  the  loser  has  little  ability  to  suggest  that  it  might  have  won  if  the  government  had 
not  made  some  error — that  is,  to  have  a  protest  sustained.  This  latter  option  becomes  more 
attractive  as  sophisticated  protests  become  more  likely.  When  they  are  likely,  the  Air  Force  can 
limit  discussions  by  giving  offerors  fewer  opportunities  to  revise  their  proposals  in  response  to 
Air  Force  feedback.  Telling  offerors  that  the  Air  Force  will  limit  discussions  and  then  follow¬ 
ing  through  could  induce  offerors  to  proffer  their  true  FPR  earlier  and  preserve  differences  that 
could  strengthen  the  Air  Force’s  hand  when  inevitable  protests  occur. 


14  As  we  explain  below,  the  MER  can  be  quite  helpful  in  a  system  program  review,  where  total  expected  LCC  deserves  close 
attention.  It  is  relevant  to  a  source  selection  only  if  it  helps  discriminate  among  sources.  The  Air  Force  use  of  the  MER  in 
the  CSAR-X  source  selection  offered  no  such  discriminatory  power. 
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The  draft  RFPs  for  the  new  round  of  the  KC-X  source  selection  appears  to  strike  a  balance 
that  reflects  many  of  these  observations.13  For  example,  it  relies  much  more  heavily  on  pass/ 
fail  evaluations  than  the  first  round.  It  has  dramatically  reduced  the  number  of  discriminators 
included  in  the  trade  space.  And  it  has  presented  precise  information  on  the  relative  value  the 
Air  Force  will  place  on  these  discriminators.  Each  of  these  changes  appears  to  be  a  step  in  the 
right  direction. 


13  Robert  A.  Burton,  director  of  the  White  House  Office  of  Federal  Procurement  Policy  under  the  last  administration, 
has  argued  that  such  an  approach  is  “inconsistent  with  the  principal  reasons  the  Congress  enacted  [the  Weapons  Systems 
Acquisition  Reform  Act  of  2009]  WSARA.”  (See  Robert  A.  Burton,  “Analysis  of  KC-X  Tanker  Draft  RFP  for  Consistency 
with  the  Weapons  Systems  Reform  Act  of  2009,”  white  paper,  Washington,  D.C.:  Venable,  L.L.P.,  October  19,  2009.)  We 
disagree.  Our  reading  of  WSARA  tells  us  that  it  favors  a  broad  trade  space  early  in  a  program — for  example,  in  the  require¬ 
ments  determination  phase.  WSARA  seeks  a  degree  of  maturation  in  programs  entering  development  that  by  definition  has 
already  dramatically  narrowed  the  trade  space  based  on  earlier  analysis.  By  the  time  a  program  managed  under  WSARA- 
type  guidance  reaches  source  selection,  we  would  not  be  at  all  surprised  to  see  the  kind  of  constrained  trade  space  spelled 
out  in  the  most  recent  KC-X  RFP. 
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Quantitative  Cost  Methods  Give  GAO 
Openings  to  Question  Reasonableness 

•  Life  cycle  costs  (LCC)  will  remain  important  to 
USAF  source  selections. 

-  Explicit  formulas  invite  close  questioning  of 
assumptions  about  the  future. 

-  Stoplight  assessment  appears  to  leave  more 
latitude  for  military  judgment  if  USAF 
documents  evaluation  decisions. 

*  Caution:  The  requirement  to  estimate  LCC  at  DAB 
Milestone  B  will  remain. 

-  USAF  must  clearly  explain  the  basis  for  any 
difference  between  this  estimate  and  estimates 
used  in  a  corresponding  RFP. 

-  Distinct  cost  estimates  can  increase  demands 
on  limited  resources 
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Quantitative  Cost  Methods  Give  GAO  Openings  to  Question  Reasonableness 

Three  of  the  ten  grounds  for  protest  that  GAO  sustained  in  the  CSAR-X  and  KC-X  source 
selections  involved  GAO’s  assessment  of  Air  Force  LCC  estimation  practices.16  LCC  will  con¬ 
tinue  to  be  important  in  future  Air  Force  source  selections  as  a  potentially  potent  discrimina¬ 
tor  among  proposals.  So  it  will  likely  be  important  for  the  Air  Force  to  adjust  its  treatment  of 
LCCs  in  source  selection. 

The  Air  Force  has  already  changed  policy  to  require  high-level  approval  before  a  source 
selection  uses  MPLCC  as  a  criterion.17  Instead,  the  Air  Force  will  follow  the  lead  of  Naval  Air 
Systems  Command  (NAVAIR)  and  focus  cost  estimates  in  a  source  selection  on  the  instant 
contract — in  effect,  on  the  price  that  a  proposal  offers  for  the  deliverables  covered  by  the  source 
selection.  To  reflect  likely  program  costs  beyond  the  deliverables  covered  by  the  contract,  this 
approach  addresses  future  supportability  as  a  subfactor  in  the  technical  proposal  and  frames 
supportability  in  terms  of  such  engineering  characteristics  as  mean  time  between  failure,  mean 
time  to  repair,  system  availability  or  maintainability,  and  the  like.  GAO  has  accepted  this 
approach  in  NAVAIR  source  selections  for  major  systems.  This  approach  does  not  actually 


16  One  of  these,  the  first  sustainment  in  the  CSAR-X  program,  ultimately  turned  on  a  different  issue — the  Air  Force’s  fail¬ 
ure  to  execute  the  approach  to  life  cycle  costing  implied  by  its  RFP.  But  GAO’s  discussion  of  this  error  gave  close  attention 
to  the  Air  Force’s  approach  to  LCC  estimation  and  suggested  that  a  reasonable  person  would  have  expected  the  Air  Force 
to  take  a  different  approach  than  the  one  the  Air  Force  used  in  evaluation.  The  two  sustainments  from  the  KC-X  program 
explicitly  reject  how  the  Air  Force  estimated  specific  elements  of  LCC,  calling  them  “unreasonable.” 

17  U.S.  Air  Force,  2009a. 
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attempt  to  estimate  future  cost.  Rather,  it  would  allow  the  Air  Force  to  give  a  proposal  credit 
for  engineering  characteristics  likely  to  lead  to  lower  future  costs. 

If  the  Air  Force  really  cares  about  LCCs,  why  should  it  use  engineering  proxies  for  future 
costs?18  The  accounts  used  to  justify  estimates  of  the  MPLCC  appear  to  clarify  Air  Force 
assumptions  about  an  inherently  uncertain  future  and  to  monetize  all  assumptions  with  a 
degree  of  precision  that  invites  GAO  judgments  about  reasonableness.  The  completeness  and 
quantitative  nature  of  LCC  accounts  require  stark  Air  Force  statements  about  inherently  sub¬ 
jective  assumptions  about  a  highly  uncertain  future.  By  contrast,  a  supportability  subfactor 
can  weigh  engineering  inputs  about  all  aspects  of  future  system  support  without  specifying  in 
advance  precisely  how  the  Air  Force  will  combine  information  about  different  inputs  to  yield 
a  final  evaluation  for  the  subfactor.  Put  another  way,  subjective  military  judgment  appears  to 
be  better  than  precise  cost  accounts  as  a  way  to  address  factors  about  which  great  uncertainty 
persists.  The  precision  required  by  cost  estimates  may  create  a  false  sense  of  certainty  about  the 
future  that  can  invite  questions  about  how  reasonable  it  is  for  the  Air  Force  to  use  any  exact 
assumption  offered.  Without  asking  whether,  objectively  thinking,  it  is  better  to  use  precise 
LCC  estimates  or  subjective  military  judgment  to  evaluate  statements  about  the  future,  GAO 
decisions  in  bid  protest  reviews  appear  to  display  this  kind  of  reasoning. 

As  the  Air  Force  is  moving  toward  an  approach  to  evaluating  the  future  supportability 
of  new  systems  in  source  selection  more  qualitatively,  it  should  keep  in  mind  that  the  DAB 
program  review  process  will  continue  to  require  quantitative  estimation  of  LCCs.  This  pro¬ 
cess  approaches  its  Milestone  B  at  almost  exactly  the  same  time  that  the  corresponding  source 
selection  approaches  a  final  decision.  That  is,  in  the  future,  the  Air  Force  will  use  one  estimate 
of  cost  in  its  source  selection  process  and,  nearly  simultaneously,  a  different  estimate  of  cost 
in  its  program  review  process.  When  it  does  this,  it  must  ensure  that  this  difference  does  not 
confuse  GAO. 

In  the  past,  GAO  did  not  see  the  program  review  estimate,  which  was  broadly  under¬ 
stood  to  be  pre-decisional  and  not  suitable  for  release  beyond  DoD.  Nonetheless,  in  several 
recent  source  selections,  GAO  has  obtained  access  to  this  estimate  in  camera  and  used  it  to 
help  assess  the  reasonableness  of  the  cost  estimates  used  in  a  source  selection.  This  is  likely  to 
happen  again  in  the  future.  When  it  does,  future  Air  Force  requests  for  proposal  should  clearly 
explain  that  different  definitions  of  cost  are  appropriate  for  the  two  different  decision  processes 
and  explain  why  the  source  selection  uses  one  while  the  program  review  uses  another.  This  will 
prevent  some  future  GAO  official  from  ruling  that  the  cost  estimate  in  a  source  selection  is 


18  An  irony  of  the  CSAR-X  source  selection  is  that  GAO  misread  Air  Force  requests  for  information  on  supportability  as 
evidence  that  the  Air  Force  cared  significantly  about  differences  among  proposals  in  future  support  costs.  In  the  CSAR-X 
source  selection,  Air  Force  interest  in  supportability  stemmed  from  concerns  that  the  legacy  system  that  a  CSAR-X  system 
would  replace  could  not  service  new  mission  requests  quickly  and  reliably  enough  because  its  system  availability  rate  was 
unacceptably  low.  The  Air  Force  wanted  a  CSAR-X  system  to  display  high  availability  to  reduce  the  time  between  receipt 
of  a  call  for  support  and  execution  of  a  support  mission,  even  if  higher  availability  involved  higher  future  costs.  Because  the 
RFP  did  not  indicate  how  the  Air  Force  planned  to  use  data  on  supportability,  GAO  misread  the  Air  Force’s  intent  in  the 
source  selection.  All  of  this  should  remind  us  that,  even  though  supportability  can  serve  as  a  proxy  for  LCC,  that  is  not  the 
only  way  it  can  be  used.  To  avoid  misunderstanding,  the  Air  Force  should  be  as  clear  as  possible  about  how  it  plans  to  use 
inputs  on  supportability  or  any  other  factor. 
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“unreasonable”  because  it  differs  from  the  estimate  the  Air  Force  uses  in  an  entirely  separate 
but  closely  related  decision  process.19 

Given  the  shortage  of  trained,  experienced  Air  Force  acquisition  personnel,  developing 
and  maintaining  two  separate  cost  estimates  at  precisely  the  same  time  could  strain  resources 
more  than  using  a  single  estimate  to  support  both  the  source  selection  and  the  program  review. 
But,  in  fact,  costs  relevant  to  the  instant  contract  in  the  source  selection  are  a  subset  of  the  total 
costs  relevant  to  the  program  review.  So  the  only  additional  effort  required  to  maintain  two 
separate  cost  estimates  is  to  be  clear  about  which  LCCs  to  associate  with  the  instant  contract. 
To  the  extent  that  the  Air  Force  uses  a  WBS  to  build  cost  estimates  from  the  bottom  up,  costs 
relevant  to  the  instant  contract  should  occupy  a  distinct  portion  of  the  WBS  and  so  be  easy  to 
identify.  Using  the  WBS  to  distinguish  the  two  cost  estimates,  then,  should  help  explain  the 
differences  between  two  separate  estimates  and  help  clarify  for  GAO  why  the  two  estimates 
differ. 


19  For  example,  the  difference  in  the  costs  relevant  to  system  program  review  and  source  selection  appears  to  have  confused 
Senator  John  McCain,  who  asked  Secretary  of  Defense  Gates,  “will  the  [KC-X]  source-selection  authority  assess  most  prob¬ 
able  life-cycle  cost  (MPLCC)”  (John  McCain  [Senator],  Letter  to  the  Honorable  Robert  M.  Gates,  U.S.  Senate  Committee 
on  Armed  Services,  Washington,  D.C.,  October  29,  2009).  A  full  assessment  of  MPLCC  is  not  required  to  discriminate 
between  the  two  offerors  for  the  KC-X,  because  many  costs  do  not  depend  on  which  system  the  Air  Force  chooses. 
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GAO  Concerns  About  Cost  Estimates  in 
CSAR-X,  KC-X  Were  Well-Founded 

•  GAO’s  judgments  about  “reasonableness”  appear 
to  overstep  focus  on  procedure,  but 

•The  substance  of  GAO  concerns  appears  solid. 

•  Examples: 

-  MER-based  estimates  of  O&S  costs  are  not 
good  predictors  if  USAF  plans  to  revise  MER 
when  a  new  system  design  is  mature  (CSAR-X). 

-  Factors  relevant  to  total  EMD  cost  growth  can 
say  little  about  growth  in  nonrecurring 
engineering  costs  within  EMD  (KC-X). 

•  GAO  observations  could  help  USAF  cost  analysts. 
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GAO  Concerns  About  Cost  Estimates  in  CSAR-X  and  KC-X  Were  Well- 
Founded 

GAO’s  assessments  of  Air  Force  cost  estimation  in  these  two  source  selections  look  exception¬ 
ally  intrusive.  They  basically  say  that  GAO  does  not  approve  of  how  the  Air  Force  uses  formal 
analysis  to  support  its  decisions.  GAO  justified  its  sustainments  on  cost  issues  in  the  KC-X 
source  selection,  not  by  reference  to  precedents  set  by  earlier  GAO  bid  protest  decisions  but 
by  reference  to  a  technical  handbook  on  cost  estimation.20  Such  a  justification  has  been  quite 
exceptional  in  the  recent  GAO  bid  protest  decisions  we  examined  in  the  course  of  this  project. 
The  Air  Force  decision  to  move  away  from  quantifying  future  costs  in  source  selection  should 
limit  its  exposure  to  such  GAO  judgments.  But  Air  Force  evaluation  of  costs  associated  with  an 
instant  contract  will  continue  to  raise  similar  issues.  What  can  the  Air  Force  learn  from  GAO’s 
discussion  of  cost  issues  in  the  CSAR-X  and  KC-X  decisions? 

First,  GAO’s  discussion  of  cost  estimation  in  the  three  grounds  for  protest  sustainment 
that  turned  on  cost  issues  was  well  informed.  In  each  case,  GAO  applied  sound  costing  prin¬ 
ciples  to  produce  a  better  approach  to  cost  estimation  than  the  Air  Force  had  used. 

For  example,  in  the  CSAR-X  program,  GAO  questioned  why,  if  the  Air  Force  planned 
to  adjust  its  MER  in  response  to  early  experience  with  any  new  CSAR  system,  it  was  appro¬ 
priate  to  use  the  MER  for  a  legacy  system  that  in  all  likelihood  would  have  different  support 
requirements  from  any  new  system.  The  Air  Force  has  used  an  MER  for  decades  to  estimate 


20  U.S.  Government  Accountability  Office,  2007a. 
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the  support  costs  of  new  systems,  in  program  reviews,  and  in  source  selections.21  When,  in 
response  to  GAO’s  first  sustained  protest  in  the  CSAR-X  program,  the  Air  Force  clarified 
that  that  is  precisely  what  it  intended  to  do — to  estimate  MPLCC  of  new  systems  based  on 
the  manning  levels  defined  by  the  MER  for  a  legacy  system,  GAO  backed  off  and  accepted 
this  clarification.  But  GAO’s  initial  observation  was  correct — the  Air  Force’s  approach,  now 
clearly  stated  in  the  amended  RFP,  was  not  the  approach  a  reasonable  outside  observer  would 
expect  the  Air  Force  to  use  in  a  source  selection.  At  a  minimum,  because  it  provided  no  abil¬ 
ity  to  discriminate  between  the  proposals,  it  was  clear  that  this  approach  added  no  value 
in  the  source  selection.  Although  GAO  acceded  to  this  flawed  approach  to  costing  when 
the  Air  Force  clearly  explained  it,  the  Air  Force  would  benefit  from  learning  from  GAO’s 
initial  reservations.  Doing  so  would  improve  Air  Force  decisionmaking  in  future  source 
selections — and  perhaps  in  other  decision  processes  that  use  a  similar  approach  to  cost 
estimation. 

GAO  was  more  intrusive  in  the  KC-X  source  selection.  It  identified  two  cases  where  the 
Air  Force  used  a  flawed  approach  to  estimating  future  costs.  This  time,  it  ruled  that  these 
approaches  were  unacceptable,  whether  or  not  the  Air  Force  explained  them  clearly  in  the  RFP. 
That  is,  in  each  case,  GAO  sustained  a  protest,  not  because  of  failure  to  match  evaluation  to 
the  plan  identified  in  the  RFP  but  because  the  plan  itself  was  flawed.  To  understand  GAO’s 
view,  consider  the  second  protest  GAO  sustained  on  these  grounds  in  the  KC-X  case.  GAO 
rejected  Air  Force  use  of  historical  data  on  growth  in  total  development  and  production  costs 
to  estimate  future  growth  in  one  component  of  development  costs — nonrecurring  engineering 
costs.  Once  again,  as  intrusive  as  GAO  was  in  this  case,  its  observation  was  correct.  By  learn¬ 
ing  from  this  GAO  judgment,  the  Air  Force  could  improve  decisionmaking  in  future  source 
selections  and  elsewhere.  The  lesson  is  not  just  about  future  cost  growth,  which  the  Air  Force’s 
new  approach  to  cost  estimation  in  source  selection  should  render  irrelevant,  but  about  justify¬ 
ing  the  historical  data  used  to  assess  cost  estimates  that  will  continue  to  appear  in  future  source 
selections. 

The  good  news  is  that,  if  GAO  continues  to  take  an  intrusive  stance  on  cost  methodology — 
and  now  it  has  precedent  it  can  cite  to  do  so — there  is  a  good  chance  that  it  will  use  sound  cost 
estimation  principles  as  a  basis  for  such  intrusion.  That  will  make  it  easier  for  Air  Force  cost 
analysts  to  prepare  for  future  GAO  attention  to  their  work  in  source  selections. 


21  The  Air  Force  mans  its  in-house  maintenance  units  so  that  they  can  execute  expected  wartime  missions,  including  sig¬ 
nificant  surges.  Having  the  assured  ability  to  execute  the  wartime  mission  has  traditionally  been  more  important  to  the  Air 
Force  than  the  cost  of  doing  this.  Using  data  from  the  MER  to  calculate  future  O&S  costs  can  be  viewed  as  a  way  of  placing 
higher  priority  on  executing  the  wartime  mission  than  on  savings  in  support  costs  claimed  for  nonmilitarized  commercial- 
based  helicopters.  That  said,  the  Air  Force  adjusts  the  MER  for  a  weapon  system  over  time  to  reflect  its  actual  experience 
with  the  system.  If  the  helicopters  offered  in  the  competition  operate  as  claimed,  the  Air  Force  knows  that  it  will  adjust  its 
MER — and  any  cost  estimates  based  on  it — to  reflect  that  realized  performance  of  the  winning  helicopter.  If  the  Air  Force 
accepts  the  claims  of  the  winning  offeror,  it  can  reasonably  expect  that  the  support  costs  for  the  winner  will  be  lower  than 
the  support  costs  implied  by  the  MER  for  the  legacy  system. 
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Minute  Details  Led  GAO  to  Sustain 
Three  Protests  in  CSAR-X,  KC-X 

•  CSAR-X  #1 :  One  sentence  in  an  attachment  to  RFP 
Section  L  let  GAO  redefine  support  costs 

•  KC-X  #2:  Two  sentences  in  disparate  parts  of  RFP 
let  GAO  reject  credit  for  higher  fuel  offload  capacity 

•  KC-X  #3:  One  evaluation  notice,  incompletely 
closed,  let  GAO  question  whether  winner  met  a 
basic  requirement 

•  Two  conclusions  are  possible: 

1.  Perfection  is  impossible;  errors  are  inevitable. 

2.  GAO  makes  the  rules;  USAF  can  learn  them. 

s|D 
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Minute  Details  Led  GAO  to  Sustain  Three  Protests  in  CSAR-X  and  KC-X 

Three  of  the  ten  grounds  for  protest  that  GAO  sustained  in  the  CSAR-X  and  KC-X  source 
selections  involved  small  errors  that  would  be  very  hard  to  detect  without  a  minutely  detailed 
review  of  the  RFP  and  evaluation  process.  These  errors  created  ambiguities  that  GAO  con¬ 
cluded  were  significant  enough  to  potentially  affect  the  outcome  of  the  source  selection.  A 
quick  review  of  them  illustrates  why  they  are  so  hard  to  catch. 

The  first  sustainment  in  the  CSAR-X  program  resulted  from  the  wording  in  one  sentence 
in  an  attachment  to  Section  L,  which  defines  the  inputs  the  Air  Force  wants  in  each  proposal, 
not  Section  M,  which  explains  how  the  Air  Force  will  evaluate  inputs  from  proposals.22  That 
sentence  states  that  the  “primary  purpose”  of  the  attachment  is  to  “capture  all  relevant  CSAR-X 
Operating  and  Support  (O&S)  costs.”  GAO  read  this  table  to  indicate  that  the  evaluation  pro¬ 
cess  would  consider  proposal-specific  information  on  O&S  costs  in  its  assessment  of  cost.  The 
attachment  explained  that  “Unit  Mission  Personnel,  Training  Munitions,  and  Indirect  Sup¬ 
port  will  be  calculated  by  the  government  team,”  but  the  RFP  never  explained  how  that  would 
happen.  GAO  inferred  from  language  throughout  the  RFP  that  a  “reasonable  person”  would 
expect  the  Air  Force  to  use  inputs  from  the  proposals  to  make  these  calculations.  Although 
GAO  drew  this  inference  from  language  throughout  the  RFP,  its  discussion  treated  this  one 
sentence  as  the  linchpin  that  made  such  an  inference  reasonable.  When  the  Air  Force  clearly 
explained  how  the  government  would  execute  the  calculations  in  question  in  its  amended  RFP, 


22  U.S.  Air  Force,  2005,  §  L,  Attachment  13. 
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GAO  withdrew  its  reservations,  even  though,  as  noted  above,  it  thought  the  method  the  Air 
Force  would  use  to  make  the  calculations  was  substantively  inappropriate  for  the  task  at  hand. 

The  second  sustainment  that  GAO  listed  for  the  KC-X  source  selection  similarly  resulted 
from  ambiguity  that  resulted  from  just  a  few  words  in  the  RFP — two  sentences  in  the  SRD 
and  one  sentence  in  Section  M.  The  SRD  set  a  KPP  threshold  (a  mandatory  minimum  require¬ 
ment)  for  fuel  offload  versus  unrefueled  range  and  a  KPP  objective  that  the  offerors’  “aircraft 
should  be  capable  of  exceeding”  the  threshold.23  Section  M  then  stated  that  “no  consideration 
will  be  provided  for  exceeding  KPP  objectives.”24  Taken  together,  these  three  statements  clearly 
say  that,  once  an  offeror  achieves  the  threshold  requirement  for  fuel  offload  versus  unrefueled 
range,  the  Air  Force  will  give  an  offeror  no  consideration  for  exceeding  the  threshold.  This  was 
clearly  not  the  intent  of  the  Air  Force,  but  the  RFP  clearly  stated  that  this  is  what  the  Air  Force 
would  do. 

To  catch  either  of  these  errors,  someone  would  have  to  read  the  entire  RFP  in  all  of  its 
detail,  hold  all  of  that  detail  in  his  or  her  head  at  the  same  time,  and  see  the  implications  of 
statements  in  unexpected  places  or  unexpected  juxtapositions  of  various  statements  in  different 
places.  It  is  worth  noting  that  GAO  does  not  have  the  resources  to  do  this.  It  relies  on  protest¬ 
ers  to  bring  these  inconsistencies  to  its  attention  for  its  consideration.  Once  it  has  all  the  detail 
placed  before  it  in  an  orderly  fashion,  however,  its  conclusions  become  relatively  easy  to  justify 
and  document. 

The  third  error  is  a  bit  different.  This  is  the  third  ground  for  sustainment  that  GAO  listed 
for  the  KC-X  source  selection.  The  Air  Force  told  Northrop  Grumman  twice  during  discus¬ 
sions  that  its  “initially  identified  maximum  operational  airspeed  .  .  .  would  not  be  sufficient 
under  current  Air  Force  overrun  procedures  to  achieve  required  overrun  speeds  ...  for  various 
fighter  aircraft.”25  If  the  Northrop  Grumman  proposal  could  not  achieve  this  requirement,  it 
would  be  technically  unacceptable.  The  Air  Force  ultimately  accepted  Northrop  Grumman’s 
proposed  solution  as  satisfying  this  KPP  threshold  but  never  documented  the  basis  for  its 
acceptance.  Boeing  complained  that  the  Northrop  Grumman  solution  did  not  appear  to  meet 
the  threshold  requirement.  During  the  protest  proceedings,  the  Air  Force  could  not  rebut  this 
complaint. 

GAO’s  decision  discusses  this  protest  at  considerable  length  and  ultimately  concludes 
that  very  serious  doubts  exist  about  whether  Northrop  Grumman  can  meet  the  threshold.  As 
a  starting  point  for  our  own  analysis,  we  take  as  given  the  Air  Force’s  argument  that  Northrop 
Grumman  could  meet  the  threshold.  Whether  Northrop  Grumman  could  or  not  does  not 
affect  our  argument.  Even  if  the  Air  Force  was  correct  in  its  evaluation,  it  erred  in  its  docu¬ 
mentation  of  that  evaluation  and  its  defense  of  it  in  GAO  hearings  on  the  question.26  The  more 
basic  error  was  the  failure  to  close  out  the  EN  that  the  Air  Force  opened  when  it  first  queried 
Northrop  Grumman  about  this  issue.  The  KC-X  program  office  used  a  standard  information 
management  system — EZ  Source — to  track  the  status  of  all  ENs.  Despite  the  presence  of  that 
system,  the  program  office  failed  to  see  that  it  had  never  explained,  in  the  record,  the  basis 


23  U.S.  Air  Force,  2007,  SRD  §  3.2. 1.1. 1.1  and  §  3.2. 1.1. 1.2. 

24  U.S.  Air  Force,  2007,  §  M. 2. 2. 1.1. a. 

25  U.S.  Government  Accountability  Office,  2008. 

26  If  the  Air  Force’s  judgment  on  this  issue  was  not  correct,  it  faces  a  more  serious  problem  than  those  we  focus  on  here. 
That  is  a  basic  substantive  failure  that  none  of  the  solutions  we  olfer  here  can  address. 
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for  its  judgment  that  the  issue  was  resolved.  The  KC-X  source  selection  generated  hundreds  of 
ENs,  so  it  is  not  an  easy  task  to  track  all  of  them  and  verify  that  the  evaluation  team  has  docu¬ 
mented  its  resolution  of  each  issue  in  a  substantively  sound  way — a  way  that  GAO  would  view 
as  “reasonable.”  The  failure  to  track  this  one  EN  opened  the  door  for  GAO  to  raise  questions 
about  the  evaluation  during  its  hearing.  As  noted  above,  the  Air  Force  was  unable  to  explain 
its  approach  satisfactorily  during  the  hearing.  This  ultimately  led  to  a  sustained  protest.  A  small 
error  had  large  consequences. 

Such  small  errors  ultimately  forced  the  Air  Force  to  make  large  changes  in  its  approaches 
to  the  CSAR-X  and  KC-X  programs.  Many  senior  Air  Force  personnel  we  spoke  to  take  a  dire 
lesson  from  this  observation.  Perfection  is  impossible  and  so  sustainments  are  inevitable.  Even 
if  this  is  true — almost  by  definition — we  believe  a  more  constructive  lesson  to  draw  is  that 
greater  attention  to  the  details  can  improve  the  Air  Force’s  performance.  Accepting  that  GAO 
makes  the  rules  of  the  game  is  a  first  step  toward  efforts  to  have  Air  Force  source  selection  per¬ 
sonnel  understand  those  rules  more  completely.27 


27  Again,  GAO  has  no  formal  set  of  rules  that  it  applies  to  make  its  judgments;  we  refer  here  to  the  “rules”  implicit  in  the 
body  of  decisions  that  GAO  has  made  on  bid  proposals  in  the  past. 
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Anticipated  Quality  Assurance  Measures 
Cannot  Catch  Small  Errors 

•  GAO  sustained  protests  in  CSAR-X,  KC-X  both  despite 
intense  USAF  efforts  to  catch  and  correct  errors. 

•  Inability  to  catch  small  errors  appears  to  result  from 

-  Lack  of  experienced,  trained  personnel 

-  Too  much  emphasis  on  speed 

-  Lack  of  tools  to  support  discipline.  Examples: 

•  Systematic  mapping  in  RFP  of  all  data  requested 
to  all  evaluation  events  that  use  them 

•  Systematic  tracking  and  documented  closeout  of 
each  evaluation  notice 

AND 
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Anticipated  Quality  Assurance  Measures  Cannot  Catch  Small  Errors 

One  way  to  try  to  catch  errors  of  the  kind  just  discussed  is  to  make  increased  use  of  external, 
high-level  teams  to  enhance  QA.  The  new  Air  Force  AIP  places  heavy  emphasis  on  such  teams, 
which  will  check  source  selections  at  several  critical  points  in  time.  The  CSAR-X  and  KC-X 
programs  had  very  high  visibility  and  used  external  review  processes  very  similar  to  those 
implemented  in  the  AIR  Despite  such  efforts,  they  suffered  ten  grounds  for  protest  sustained 
by  GAO  (two  in  CSAR-X  and  eight  in  KC-X).  Why?  An  examination  of  the  two  programs 
suggests  that  three  factors  contributed. 

Both  programs  had  shortages  of  properly  trained  personnel  with  enough  prior  experi¬ 
ence  in  complex  source  selections  to  prepare  them  for  handling  the  tough,  day-to-day  task  of 
scrubbing  down  an  RFP  and  then  documenting  and  controlling  evaluation  material  carefully 
to  prevent  adversarial  attorneys  representing  protesters  from  finding  residual  problems  in  the 
record.  The  two  programs  relied  heavily  on  a  small  number  of  senior  personnel  to  train  and 
mentor  the  personnel  who  actually  executed  most  of  the  work  during  the  source  selection. 
Ongoing  efforts  to  rebuild  the  organic  Air  Force  acquisition  workforce  should  help  address 
this  problem,  but  they  will  take  many  years  to  make  a  significant  difference  in  future  Air  Force 
source  selections. 

Both  source  selections  moved  too  fast.  The  KC-X  program  office  felt  pressure  from  above 
to  move  forward,  at  least  in  part  to  offset  the  considerable  delays  that  the  program  had  suf¬ 
fered  before  this  source  selection.  It  was  unsuccessful  in  its  efforts  to  push  back  against  this 
pressure.  A  number  of  people  suggested  that  excess  speed  contributed  to  errors  that  could  have 
been  avoided  if  more  time  had  been  available  to  review  decisions  thoroughly.  The  CSAR-X  put 
a  premium  on  speed,  believing  that  the  faster  the  source  selection  occurred,  the  fewer  external 
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surprises  could  occur  that  could  change  the  basic  parameters  of  the  source  selection  and  so  add 
administrative  burden  and  complexity  to  the  process.28  But  combining  speed  with  a  shortage 
of  personnel  ultimately  burned  out  the  key  personnel  in  the  source  selection.  If  the  CSAR-X 
had  continued  on  to  another  source  selection,  a  new  source  selection  team  would  probably 
have  had  to  be  assembled  to  relieve  those  who  went  through  the  first  two  rounds  of  protests. 
In  effect,  after  building  considerable  experience  in  the  context  of  the  CSAR-X  source  selec¬ 
tion,  the  Air  Force  would  have  had  to  train  a  fresh  team  and  start  over  with  similar  gaps  in  the 
experience  of  the  team.  The  recent  shift  in  Air  Force  policy  to  make  source  selections  event- 
oriented  rather  than  schedule-oriented  could  potentially  ameliorate  some  of  these  pressures  if 
the  Air  Force  can  in  fact  maintain  the  discipline  required  to  give  source  selections  the  time 
required  to  complete  appropriate  tasks.  Because  source  selections  typically  draw  in  people  with 
other  responsibilities,  pressure  is  likely  to  continue  to  get  through  them  quickly  so  that  they 
can  release  these  personnel  to  their  primary  longer-term  duties. 

Two  types  of  tools  could  have  helped  structure  activities  in  these  source  selections  in  ways 
that  made  it  easier  to  catch  small  errors  such  as  those  discussed  above.  One  tool  would  track 
each  evaluation  activity  described  in  Section  M  to  the  specific  requirement  that  this  evaluation 
addresses  and  to  each  piece  of  information  requested  in  Section  L  in  support  of  this  evaluation 
activity.  Such  a  tracking  tool  would  facilitate  consistency  checks  across  the  entire  RFP  and  help 
ensure  that  the  RFP  describes  the  specific  evaluation  methods  the  Air  Force  will  use  to  explain 
how  the  methods  inform  requirements  and  how  they  use  information.  Such  a  mapping,  in 
itself,  could  clarify  the  Air  Force’s  plans  for  evaluation  in  ways  that  could  (1)  make  it  harder 
for  offerors  to  claim  misunderstanding  about  evaluation  and  (2)  provide  a  clear  benchmark 
against  which  to  track  evaluation.  Many  Air  Force  source  selections  have  used  a  simple  matrix 
that  supports  such  tracking.  The  draft  RFP  in  the  new  round  of  the  KC-X  source  selection  uses 
one.  Neither  the  CSAR-X  nor  the  original  KC-X  source  selection  used  one. 

A  second  tool  could  track  ENs  from  their  inception  to  their  final  resolution.  An  effective 
system  of  this  kind  would  support  ongoing  reviews  of  such  questions  as  the  following:  (1)  Has 
each  question  raised  in  an  EN  been  resolved?  (2)  When  differences  of  opinion  arose  within  the 
evaluation  team  in  response  to  ENs,  were  these  differences  decisively  addressed  and  disposed 
of?  (3)  Does  the  record  contain  sufficient  documentation  to  justify  resolution  of  the  question? 
(4)  Does  the  record  include  any  information  not  required  to  track  evaluation  and  justify  final 
decisions?  If  so,  why  is  it  there?  Air  Force  programs  often  rely  on  EZ  Source,  a  standard  infor¬ 
mation  management  system,  to  check  regularly  the  status  of  documentation  against  GAO- 
informed  standards.  Both  the  CSAR-X  and  KC-X  programs  used  this  system.  Unfortunately, 
EZ  Source  is  a  cumbersome  system  that  records  actions  but  does  not  ensure  as  much  as  it 
could  appropriate  execution  of  these  actions.  In  the  lessons  learned  exercise  conducted  at  the 
close  of  the  CSAR-X  program,  for  example,  EZ  Source  was  the  dominant  object  of  complaints 
about  how  the  source  selection  proceeded.  Whatever  system  is  used,  it  cannot  simply  be  used 
to  check  boxes.  A  proactive  awareness  of  how  the  record  created  by  this  process  will  look  to 
outsiders  should  lead  to  more  discipline  in  the  maintenance  of  the  records  required  to  justify 
decisions  and  the  maintenance  of  only  these  records. 


28  They  envisioned  themselves  operating  within  a  classic  Observation-Orientation-Decision-Action  loop  in  which  they 
would  keep  their  decision  cycle  as  short  as  possible  relative  to  those  in  other  processes  that  could  affect  the  source  selection 
(John  R.  Boyd,  “Discourse  on  Winning  and  Losing,”  briefing,  1976). 
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Only  People  Inside  a  Source  Selection  Can 
Assure  That  It  Gets  the  Small  Things  Right 

•  Rebuild  the  acquisition  workforce.  This  will  take  a 
long  time. 

•  In  the  meantime 

-  Use  practical  training  to  substitute  for  currently 
missing  experience 

-  Back  up  junior  personnel  with  skilled  technical 
personnel 

-Systematize  document  control;  train  personnel 
to  retain  only  what  is  needed 
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Only  People  Inside  a  Source  Selection  Can  Assure  That  It  Gets  the  Small 
Things  Right 

As  noted  above,  the  Air  Force  is  rebuilding  its  organic  acquisition  workforce,  but  that  will 
take  time.  The  hiatus  in  hiring  during  the  1990s  has  left  the  Air  Force  with  a  serious  short¬ 
age  of  contracting  personnel  with  10  to  20  years  of  experience.29  This  is  the  cohort  that  has 
traditionally  carried  the  heaviest  load  in  executing  day-to-day  acquisition  activities  of  the  kind 
experienced  in  source  selections.  Entry-level  hiring  today  will  not  refill  the  ranks  of  this  group 
for  another  decade  or  more.  A  significant  portion  of  the  cohort  with  more  than  20  years  of 
experience,  which  the  Air  Force  has  traditionally  relied  on  heavily  for  training  young  people, 
is  nearing  retirement  age.  Given  these  demographic  facts,  what  can  the  Air  Force  do  today  to 
enhance  the  capabilities  of  the  people  within  source  selections  who  ultimately  must  build  qual¬ 
ity  into  a  source  selection  to  avoid  sustained  protests?  Our  discussions  with  personnel  around 
the  Air  Force  supported  the  following  findings. 

Most  observers  agree  that  the  best  way  to  learn  how  to  do  source  selections  is  to  partici¬ 
pate  in  them.  Real-world  experience  teaches  by  forcing  personnel  to  make  complex  judgments 
in  an  ill-defined  decision  environment.  So  the  best  form  of  training  for  a  major  source  selection 
is  real-world  experience  in  a  smaller  source  selection  where  errors  will  cost  the  Air  Force  less. 
One  reason  the  Air  Force  is  experiencing  its  current  shortage  of  personnel  with  source  selec¬ 
tion  experience  is  because  it  has  been  conducting  fewer  complex  source  selections  in  recent 
years  than  in  the  past.  To  some  degree,  case-based  training  and  role-playing  exercises  tailored 


29  Kayes,  2008,  Chart  29. 
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to  the  circumstances  in  a  specific  new  source  selection  can  help  substitute  for  experience  in  real 
source  selections.  They  cannot  replace  such  experience,  but  they  offer  low-risk  environments  in 
which  participants  can  experience  the  consequences  of  making  hard  decisions  in  a  structured 
environment.  They  are  time-consuming  and  costly,  in  terms  of  the  time  of  the  participants  and 
the  cost  of  the  trainers.  But  if  coordinated  closely  with  the  terms  of  an  upcoming  source  selec¬ 
tion,  they  can  help  a  small  number  of  coaches  prepare  new  players  for  the  roles  they  will  play. 

Coaches  can  continue  to  support  the  players  through  the  course  of  a  source  selection  by 
watching  over  their  shoulders  as  they  go  through  their  day-to-day  activities.  A  coach  or  mentor 
on  site  allows  quick  detection  of  and  recovery  from  errors  as  they  occur.  Presence  on  site  keeps 
the  coach  knowledgeable  about  the  status  of  the  source  selection  and  so  well  prepared  to  detect 
trouble  and  help  remedy  it  when  it  occurs.  A  coach  might  even  participate  occasionally  if 
things  get  too  demanding.  But  mainly,  a  coach  should  work  to  stretch  his  or  her  own  capabili¬ 
ties  by  transferring  them  to  others  doing  the  heavy  work. 

Proper  document  management  occurs  every  day.  The  best  way  to  judge  the  information 
used  to  support  a  decision  is  to  consult  the  documents  created  as  the  decision  occurred.  As  a 
result,  GAO  relies  heavily  on  documentation  generated  contemporaneously  with  the  decisions 
involved  and  gives  little  weight  to  new  documents  generated  after  a  protest  raises  a  question. 
Well-crafted,  concise  documentation  can  dispose  of  questions  quickly  without  much  verbiage. 
Poorly  focused  documentation  crafted  without  a  clear  purpose  in  mind  can  generate  as  many 
questions  as  it  resolves.  Finding  the  right  balance  of  creating  just  enough  to  justify  a  decision 
in  a  compelling  way  and  not  so  much  that  irrelevant  issues  come  to  light  must  happen  in  real 
time.  A  good  document-tracking  system  and  good  coaching  should  facilitate  inherently  on- 
the-job  training  that  at  the  same  time  generates  the  primary  information  that  will  inform  any 
bid  protest. 
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The  Right  Kind  of  External  Quality 
Assurance  Can  Help 


Timely 

Find  errors  early  enough  to  allow 
corrections 

Well- 

informed 

Read  advisors  in  before  they  arrive 
to  avoid  increasing  the  burden  on 
overworked,  inexperienced  staff 

Technical 

Choose  advisors  technically 
sophisticated  enough  to  anticipate 
fine  distinctions  that  GAO  makes 

Hands-On 

Devil  is  in  the  details  when 
justifying  an  evaluation  plan  and 
matching  it  to  evaluation  execution 
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The  Right  Kind  of  External  Quality  Assurance  Can  Help 

As  the  Air  Force  goes  forward  wirh  its  plans  to  use  external  QA  teams  more  aggressively,  the 
CSAR-X  and  KC-X  programs  offer  insights  into  how  to  do  this  better  than  the  Air  Force  did 
in  those  programs.  Personnel  associated  with  both  programs  believe  that  external  oversight  is 
more  likely  to  add  value  if  it  displays  four  basic  characteristics. 

First,  it  is  timely.  Any  oversight,  internal  or  external,  should  occur  quickly  enough  after 
an  action  occurs  to  fix  any  errors  embodied  in  that  action — whether  that  be  preparation  of 
the  RFP,  management  of  ENs,  execution  of  evaluation,  or  justification  of  final  decisions.  For 
example,  once  the  requests  for  proposal  were  finalized  in  both  source  selections,  both  program 
offices  found  themselves  committed  to  conducting  evaluations  that  differed  from  those  they 
had  intended  to  conduct  because  this  realization  came  too  late. 

Second,  it  is  well  informed.  When  external  advisors  arrive  as  short-term  visitors,  they 
should  arrive  well  informed  about  the  issues  in  the  source  selection  and  the  status  of  the  source 
selection.  They  have  a  limited  time  on  site  to  add  value;  the  personnel  working  within  the 
source  selection  have  limited  time  to  learn  from  them  while  continuing  their  primary  tasks  in 
the  source  selection.  Asking  already  overworked  staff  to  withdraw  from  their  ongoing  source 
selection  responsibilities  to  educate  visitors,  who  might  be  somewhat  disengaged  by  their  off¬ 
site  status  to  begin  with,  can  easily  cost  more  than  the  value  it  adds. 


128  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and  Options 


Third,  it  is  technical.30  It  is  well  informed  about  the  technical  issues  relevant  to  GAO’s 
oversight  of  source  selection  decisions.  It  is  current  on  recent  lessons  learned.  It  can  engage 
junior,  less-experienced  staff  on  the  working  level  with  detailed  support  on  specific  issues.  It 
does  not  emphasize  high-level,  graybeard  strategic  thinking  unless  the  resulting  advice  can 
come  early  enough  in  the  source  selection  to  affect  its  overall  strategy.  GAO  emphasizes  fine 
points.  The  review  should  mainly  respond  to  that  emphasis. 

Finally,  it  is  hands-on.  It  addresses  specific  decisions  and  specific  documentation  in  near 
real  time,  with  the  intent  of  assuring  relevant  decisions  and  documentation  and  training  per¬ 
sonnel  within  the  source  selection  to  generate  similarly  concrete  decisions  and  documents  after 
the  team  leaves. 

It  is  no  accident  that  such  support  is  most  likely  to  come  from  senior,  experienced  techni¬ 
cians  on-site  for  the  duration  of  the  source  selection.  External  QA  teams  cannot  replace  such 
on-site  expertise.  They  can  offer  a  different  set  of  eyes  with  perhaps  a  slightly  different  interpre¬ 
tation  of  what  to  watch  for.  They  can  offer  input  from  expertise  that  is  too  costly — in  terms  of 
its  budgetary  costs  and  its  opportunity  costs — to  commit  to  a  source  selection  full  time.  But 
at  the  end  of  the  day,  even  this  higher-level  support  coming  from  a  distance  will  be  most  likely 
to  add  net  value  if  it  is  timely,  well-informed,  technical,  and  hands-on — just  like  the  coaches 
on  site. 


30  In  this  context,  we  do  not  use  “technical”  to  suggest  that  an  activity  is  associated  with  technology  or  engineering.  Rather, 
“technical  support”  stringently  applies  to  specialized  knowledge  about  each  of  the  functions  relevant  within  a  source  selec¬ 
tion  in  a  precise,  detailed,  and  up-to-date  manner  well  attuned  to  the  facts  at  hand. 
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Potential  Changes  in  Air  Force  Policy  or  Practice 


Road  Map 

*  How  source  selections  and  bid  protests  work 

*  Patterns  for  bid  protests  during  FY2000-08 

*  Protests  sustained  in  CSAR-X,  KC-X  source  selections 

*  Synthesis  of  findings 

*  Potential  changes  in  policy  or  practice 

-  Defensive  source  selection  tactics 

-  Defensive  source  selection  design 

-  Data  on  protests  and  “realized  prejudice”  in  protests 

'ID 
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The  last  chapter  raised  a  broad  range  of  potential  specific  changes  in  Air  Force  policy 
and  practice.  This  chapter  brings  these  potential  changes  together  in  one  place,  organized  as 
(1)  potential  changes  in  source  selection  tactics,  (2)  potential  changes  in  source  selection  design, 
and  (3)  collection  of  data  on  source  selection  outcomes  to  clarify  the  costs  that  inappropriate 
GAO  protest  sustainments  impose  on  the  Air  Force. 

The  Air  Force  can  change  tactics  without  changing  the  design  of  a  source  selection. 
Changing  the  design  of  a  source  selection  adds  costs  early  in  a  source  selection,  but  may  poten¬ 
tially  offset  the  overall  costs  resulting  from  a  protest  by  simplifying  evaluation  and  the  choice 
of  a  source.  Even  if  the  Air  Force  cannot  offset  these  additional  costs,  better  design  should  help 
it  avoid  future  corrective  actions  and  protest  sustainments.  It  could  potentially  do  any  of  these 
things  without  seeking  changes  in  regulations  or  legislation  beyond  its  control.  Better  data  on 
the  cost  of  sustainments  could  help  the  Air  Force  justify  the  investments  required  to  try  options 
discussed  under  the  first  two  points.  More  broadly,  new  data  could  also  potentially  help  the  Air 
Force  change  the  effective  rules  within  which  it  addresses  bid  protests.  They  would  do  this  by 
helping  the  Air  Force  build  the  case  for  adjustments  in  how  GAO  conducts  bid  protests. 
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Defensive  Tactics  Need  Not  Affect  the 
Content  of  a  Source  Selection 

*  Systematize  document  control;  retain  only  what  is 
needed 

*  Use  practical  training  to  substitute  for  currently 
missing  experience 

*  Back  up  junior  personnel  with  skilled  technical 
personnel 

*  Sharpen  QA — use  technical  personnel,  read  them  in 
early,  bring  them  early  enough  to  matter 

*  Introduce  red  teams  with  attorneys  and  other 
relevant  specialists 
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These  options  take  as  given  the  content  of  the  RFP,  suggest  ways  to  refine  the  RFP,  and 
then  evaluate  it  more  successfully. 

Better  document  control  seeks  to  ensure  that  the  basis  for  each  evaluation  decision  is 
documented  in  just  enough  depth  to  meet  GAO’s  standards  for  contemporaneous  record¬ 
keeping  and  clear  statement  of  “reasonable”  decisionmaking  and  no  more.  Similarly,  it  keeps 
“just  enough”  documentation  to  justify  the  final  choice  of  a  source.  Systematically  mapping 
requirements  to  inputs  requested  from  offerors  and  to  the  methods  in  which  the  Air  Force 
will  use  these  inputs  in  evaluation  can  clarify  exactly  what  documentation  is  needed.  A  less- 
cumbersome  document  control  system  than  EZ  Source  should  help  on  both  scores,  but  even 
more  important  is  effective  coaching  on  how  to  document  decisions  clearly  without  creating 
ambiguities  that  GAO  can  use  to  insert  its  own  judgments.  Limiting  document  flow  should 
help  the  Air  Force  focus  its  efforts  to  sustain  effective  quality  control  and  assurance  of  docu¬ 
ment  creation  and  flow. 

Until  the  Air  Force  develops  an  appropriately  experienced  acquisition  workforce,  just- 
in-time,  on-the-job  training  based  on  role-playing  can  teach  inexperienced  personnel  specific 
new  skills  that  they  can  apply  immediately  in  an  upcoming  source  selection.  Tailored  train¬ 
ing  focuses  each  person  on  the  skills  relevant  to  his  or  her  own  role  in  the  source  selection. 
Immediate  application  of  the  training  makes  it  more  likely  that  trainees  will  retain  new  skills 
acquired  in  this  way.  Role-playing  offers  personnel  an  opportunity  to  see  the  consequences 
of  errors  quickly  without  imposing  large  costs  on  the  Air  Force  when  they  make  such  errors 
during  training.  In  the  same  way  that  flight  simulators  allow  pilots  to  prepare  safely  for  high- 
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risk  situations  in  flight,  role-playing  allows  trainees  to  face  serious  challenges  and  learn  from 
them  safely.1 

Providing  experienced,  technically  skilled  coaches  to  back  up  inexperienced  personnel 
through  the  course  of  a  source  selection  makes  it  more  likely  that2 

1.  quality  control  will  detect  errors  fast  enough  to  correct  them  before  they  impose  serious 
costs 

2.  specialists  are  informed  about  the  current  status  of  the  source  selection  when  inexperi¬ 
enced  personnel  turn  to  them  for  advice 

3.  specialists  are  (ideally)  well  informed  about  GAO’s  standards  and  use  these  standards 
to  provide  backup 

4.  specialists  are  prepared  to  get  their  hands  dirty  when  personnel  need  their  help  in  hard- 
brushing  source  selection  documents  to  remove  grounds  for  sustainment  of  protests. 

In  principle,  such  coaches  could  also  provide  the  training  described  above.  Providing  external 
QA  that  features  similar  characteristics  can  ensure  this  backstop,  especially  when  oversight 
capabilities  are  too  costly  to  commit  for  the  duration  of  a  source  selection. 

Red  teams  that  include  attorneys  and  other  specialists  relevant  to  an  acquisition  can 
potentially  bring  a  new  perspective  to  oversight  that  is  more  likely  to  catch  errors  that  protest¬ 
ers  can  exploit  to  force  corrective  action  or  induce  a  sustainment.  Such  red-teaming  deliber¬ 
ately  seeks  the  weakest  spots  in  a  source  selection — in  the  RFP,  evaluation  plan,  documenta¬ 
tion  of  evaluation,  final  evaluation,  and  final  SSAC  and  SSA  briefings.  Attorneys  who  are 
uninvolved  with  the  particular  procurement  may  be  better  suited  to  such  work  as  they  may 
be  more  willing  to  critically  examine  the  judgments  made  to  date  in  the  procurement  and  the 
corresponding  protest  risks.  Private-sector  attorneys  with  significant  experience  in  bid  protests 
could  be  ideal  for  this  role,  especially  if  they  have  participated  in  such  red-teaming  for  poten¬ 
tial  protesters  in  other  procurements.  But  the  Air  Force  must  resolve  conflict  of  interest,  cost, 
and  oversight  issues  before  engaging  such  attorneys.  As  noted  above,  such  red-teaming  would 
add  administrative  costs  and  probably  add  days  to  any  source  selection  schedule.  The  Air  Force 
would  use  such  red  teams  where  it  expected  them  to  reduce  the  costs  and  schedule  delays  asso¬ 
ciated  with  protests  by  more  than  enough  to  offset  the  immediate  negative  effects  of  the  teams 
on  costs  and  schedule. 


1  Of  course,  it  is  worth  spending  a  great  deal  more  money  on  a  flight  simulator  than  on  just-in-time  training.  But  cost- 
effective,  just-in-time  training  can  be  tailored  to  the  characteristics  of  an  upcoming  source  selection.  Presumably,  more 
training  would  be  warranted  where  the  risks  to  the  Air  Force  in  a  source  selection  are  higher. 

2  Again,  in  our  usage,  technically  skilled  personnel  can  stringently  apply  specialized  knowledge  about  many  functions 
other  than  engineering. 
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Defensive  Source  Selection  Design 
Changes  Content  of  Request  for  Proposal 

•  Simplify  evaluation  criteria 

-  Reduce  number 

-  Use  more  pass-fail  requirements 

-  Focus  on  discriminators  in  trade  space 

-  Clarify  relative  priorities 

•  Simplify  cost  criteria 

-  Focus  quantitative  estimate  on  instant  contract 
-Assess  supportability  qualitatively 

-  Coordinate  with  DAB  estimates 
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Changes  in  source  selection  design  are  more  aggressive,  because  they  change  the  substan¬ 
tive  terms  the  Air  Force  uses  to  compare  proposals. 

We  endorse  the  guidance  in  the  AIP  to  simplify  evaluation  criteria.  We  believe  that  the 
importance  of  preexisting  system  designs  to  the  CSAR-X  and  KC-X  programs  led  to  some 
complications  there  that  will  be  easier  to  avoid  in  typical  future  Air  Force  acquisitions.  But  in 
any  future  acquisition,  the  guidance  of  the  AIP  is  sound.  Reduce  the  number  of  requirements 
included  in  a  source  selection.  Limit  as  many  as  possible  to  pass/fail  criteria  in  which  a  thresh¬ 
old  value  states  an  acceptable  level  of  performance.  When  a  specific  requirement  is  important 
to  the  Air  Force  and  is  likely  to  help  it  discriminate  among  proposals,  include  the  criterion  in 
the  trade  space  for  a  trade-off-based  source  selection.  Be  clear  about  the  relative  priority  that 
the  Air  Force  places  on  requirements  included  in  the  trade  space. 

The  importance  of  being  able  to  use  the  trade  space  to  discriminate  among  proposals 
should  lead  a  source  selection  to  use  a  smaller  trade  space  than  WSARA  seeks  to  promote  ear¬ 
lier  in  the  capability  planning  and  development  planning  phases  of  a  new  system  program.3 
The  Air  Force  faces  special  challenges,  as  it  moves  forward,  to  distinguish  these  two  perspec¬ 
tives.  WSARA  focuses  on  capability  and  engineering  trades  early  in  a  program,  hoping  to 
mature  the  system  under  development  considerably  before  it  reaches  DAB  Milestone  B  and 
source  selection.  Maturation  will  narrow  the  grounds  for  choice  considerably  before  source 
selection  occurs.  Narrower  grounds  for  choice  inherently  imply  a  smaller  trade  space  in  the 


3  WSARA  seeks  to  broaden  the  trade  space  considered  early  in  a  program — well  before  source  selection.  Early  consid¬ 
eration  of  a  broad  trade  space  should  allow  the  Air  Force  to  make  decisions  that  will  simplify  later  source  selection — in 
part  by  narrowing  the  trade  space  in  source  selection  to  focus  attention  on  criteria  likely  to  discriminate  among  proposals. 
Appendix  A  provides  a  short  summary  of  WSARA  and  a  discussion  of  its  implications  for  the  issues  raised  in  the  text. 
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source  selection.  The  requirements  included  in  the  source  selection  trade  space  can  then  focus 
on  attributes  important  to  the  user  community  that  are  likely  to  differ  across  proposals. 

We  also  endorse  AIP  guidance  that  encourages  a  source  selection  to  focus  cost  estima¬ 
tion  on  the  deliverables  most  directly  addressed  by  the  source  selection — those  covered  by  the 
“instant  contract.”  The  apparent  precision  of  cost  estimates  appears  to  draw  particular  pro¬ 
tester  attention  and  hence  GAO  interest.  The  inherent  uncertainty  of  future  costs  can  create 
an  appearance  of  unacceptable  ambiguity  in  a  precise  set  of  LCC  estimates.  Under  these  cir¬ 
cumstances,  using  the  technical  evaluation  to  assess  reliability  and  supportability  issues  likely 
to  affect  future  costs  appears  to  offer  a  more  reliable  way  to  address  such  uncertainty  so  as  to 
avoid  GAO  protest  sustainment.  OSD’s  DAB  program  review  requires  that  the  Air  Force  esti¬ 
mate  LCCs  at  almost  exactly  the  same  time  that  a  program  runs  its  source  selection.  To  avoid 
confusing  GAO  about  the  different  cost  estimates  used  in  these  two  processes,  the  source  selec¬ 
tion  should  clearly  explain  the  difference  and  why  the  nearer-term  estimate  used  in  the  source 
selection  is  more  appropriate  in  that  setting. 
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New  Data  Could  Help  USAF  to  Target 
Its  Attention  in  the  Future 

•  In  proactive  defense,  USAF  can  emphasize  tactics 
and  designs  above  where  risks  are  highest 

-Where  do  sophisticated  protests  occur? 

-What  costs  do  sophisticated  protests  impose  on 
USAF? 

•  Longer  term,  USAF  can  ask  if  the  “balanced”  GAO 
approach  has  in  fact  sustained  protests  only  when 
USAF  “violated  a  procurement  statute  or  regulation 
and  ...  the  violation  prejudiced  the  protester.” 

-  Prejudice:  “But  for  the  agency’s  actions,  [a 
protestor  has]  a  substantial  chance  of  receiving 
the  award.” 

-  Evidence  of  GAO  overestimate  of  prejudice  might 
provide  basis  for  regulatory  or  legislative  relief. 

ND 
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The  changes  listed  on  the  last  two  slides  can  improve  outcomes  for  the  Air  Force  in  any 
source  selection,  but  they  can  be  costly  to  execute  and  so  are  more  appropriate  in  situations 
where  it  is  more  likely  that  a  sophisticated  protest  will  occur  and  impose  costs  on  the  Air 
Force.4  We  believe  that  such  protests  will  be  more  likely  to  occur  in  large,  complex  acquisi¬ 
tions  with  high  stakes  for  the  participants,  but  we  cannot  be  much  more  specific  without  more 
information  on  when  such  protests  occur.  We  have  no  solid  information  on  the  costs  they  are 
likely  to  impose  on  the  Air  Force.  The  more  the  Air  Force  knows  about  when  they  will  occur 
and  how  they  will  likely  affect  the  Air  Force,  the  more  effectively  it  can  plan  when  to  introduce 
changes  such  as  those  discussed  above  and  how  far  to  carry  them. 

A  relatively  simple  way  to  gather  such  information  is  to  track  future  protests  sustained 
by  GAO  and  collect  information  on  their  characteristics  and  the  costs  that  they  impose.  We 
have  argued  above  that  the  Air  Force  is  as  likely  to  suffer  costs  from  corrective  actions  as  from 
sustained  protests.  But  sophisticated  protests  appear  to  be  designed  to  force  the  Air  Force  into 
a  GAO  review  where  a  well-prepared  protest  team  expects  that  it  will  outperform  an  Air  Force 
defense  well  enough  to  secure  a  GAO  sustainment  on  at  least  one  issue.  So  monitoring  the  rela¬ 
tively  small  number  of  sustained  protests  is  a  natural  place  to  start  collecting  data  to  prepare 
for  future  sophisticated  protests. 


4  Sources  of  quotes  on  the  slide:  Second  bullet:  U.S.  Government  Accountability  Office,  2006,  p.  7.  First  item  under  the 
second  bullet:  Scientific  and  Commercial  Systems  Corporation;  Omni  Corporation  (B-283160,  October  14,  1999).  Cf.  lan¬ 
guage  from  a  decision  of  the  Court  of  Appeals  for  the  Federal  Circuit,  Statistical  v.  Christopher,  102  F.3d  1577,  1581  (1996): 
“To  establish  competitive  prejudice,  a  protester  must  demonstrate  that  but  for  the  alleged  error,  there  was  a  ‘substantial 
chance  that  [it]  would  receive  an  award,  that  it  was  within  the  zone  of  active  consideration.’  CACI,  Inc.-FED.  v.  United 
States,  719  F.2nd  1567,  1574-75  (Fed.  Cir.  1983).”  Both  quoted  in  Edwards,  2006,  p.  339. 
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Over  time,  data  collected  for  this  purpose  could  potentially  serve  the  Air  Force  in  another 
way.  When  GAO  sustains  a  protest,  it  reaches  a  judgment  that  the  Air  Force  has  made  a  large 
enough  error  potentially  to  change  the  outcome  of  the  source  selection  in  question.  Is  the  error 
in  fact  large  enough  to  change  the  outcome?  When  the  Air  Force  implements  any  changes 
suggested  by  GAO,  does  the  Air  Force  then  award  a  contract  to  a  different  offeror  than  it  did 
when  the  protest  occurred?  If  it  does,  the  error  presumably  prejudiced  the  ultimate  winner, 
who  received  the  contract  only  because  the  Air  Force  corrected  its  error.  If  not — if  the  sus¬ 
tained  protest  has  no  effect  on  the  ultimate  winner — the  Air  Force  error  was  probably  not  seri¬ 
ous  enough  to  prejudice  the  offeror  whose  protest  was  sustained.5  How  often  does  this  occur? 
A  preliminary  Air  Force  review  of  recent  sustained  protests  found  that  many  sustained  protests 
do  not  lead  to  changes  in  the  winner  of  an  Air  Force  competition,  suggesting  that  GAO  has 
tended  to  misjudge  the  degree  of  prejudice  associated  with  an  Air  Force  error  in  these  protests.6 

If  more  systematic  evidence  on  how  sustainments  affect  the  ultimate  winner  of  Air  Force 
source  selections  supported  this  observation,  the  Air  Force  could  potentially  use  such  evidence, 
along  with  evidence  on  the  cost  imposed  by  such  unwarranted  sustainments,  to  inform  GAO 
and  Congress  of  the  effects  of  such  sustainments.7  As  a  fundamental  element  of  its  approach 
to  reviewing  bid  protests,  GAO  seeks  a  “balanced”  approach  that  protects  offerors  that  do 
experience  real  prejudice  in  federal  source  selections  without  imposing  undue  costs  on  other 
participants  in  these  source  selections  when  GAO  determines  that  prejudice  was  not  present.8 
The  Air  Force  could  use  evidence  of  the  kind  described  here  to  test  how  well  GAO’s  approach 
to  balance  has  worked  in  practice.  If  GAO  has  in  fact  sustained  too  many  protests  where  it 
turns  out  that  prejudice  was  not  present,  the  Air  Force  would  have  a  strong  empirical  basis  for 
arguing  that  GAO  is  not  implementing  the  balance  it  seeks  as  effectively  as  it  could. 

In  the  wake  of  the  recent,  high-visibility  sustained  protests  associated  with  the  Druyun 
scandal  and  then  in  the  CSAR-X  and  KC-X  programs,  it  is  easy  to  lose  sight  of  the  Air  Force’s 
broader  experience  with  GAO  bid  protests.  During  the  1990s,  the  number  of  unwarranted 
protests  dropped  markedly,  leaving  the  Air  Force  in  a  position  to  focus  on  protests  that  were 
more  likely  to  require  corrective  action.  Since  2001,  the  numbers  of  corrective  actions  and  of 
merit  reviews  per  1,000  contract  awards  have  slowly  dropped.  The  Air  Force  should  be  care¬ 
ful  to  protect  the  policies  and  practices  that  have  supported  this  pattern  of  steadily  improving 
performance. 

The  threat  manifested  in  the  CSAR-X  and  KC-X  programs  appears  to  be  relatively  new 
in  character  and  so  does  justify  significant  adjustments  in  policies  and  practices  in  appropriate 
circumstances.  But  we  expect  this  threat  to  present  itself  in  a  relatively  small  number  of  acqui- 


5  We  condition  these  statements,  speaking  of  “presumably  prejudiced”  and  “probably  not  serious”  because,  following  a 
protest,  the  Air  Force  can  never  simply  correct  an  error  and  rerun  precisely  the  same  competition  it  ran  earlier.  The  com¬ 
petition  following  the  protest  is  necessarily  a  new  competition  that  takes  place  in  new  circumstances.  So  the  result  of  any 
second  competition  cannot  tell  us  with  certainty  that  prejudice  was  present  in  the  first  competition.  But  the  more  often  the 
second  competition  fails  to  overturn  the  decision  of  the  first,  the  more  likely  it  is  that  material  errors  are  not  occurring  in 
the  initial  competitions.  Such  a  pattern  would  suggest  that  GAO  balances  its  decisions  too  much  in  favor  of  protesters. 

6  Kayes,  2008,  Charts  15-16. 

7  In  principle,  to  speed  development  of  a  meaningful  body  of  evidence,  the  Air  Force  could  collect  more  historical  data  as 
well  as  data  on  sustainments  in  source  selections  run  by  other  federal  agencies.  Either  approach  would  be  more  costly  than 
gathering  information  on  Air  Force  sustainments  as  they  occur.  Gathering  reliable  historical  data  well  after  the  fact  on  the 
costs  imposed  by  unwarranted  sustainments  would  be  particularly  challenging. 

8  U.S.  Government  Accountability  Office,  2009. 
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sitions — the  large,  complex  ones,  with  relatively  large  stakes  for  the  participants,  and  probably 
more  likely  when  the  participants  understand  how  to  pursue  sophisticated  protests.  Such  pro¬ 
tests  will  continue  and  could  increase  in  number  until  the  Air  Force  demonstrates  that  it  can 
effectively  counter  them.  The  Air  Force  should  focus  its  countermeasures  on  the  places  where 
the  threat  is  greatest.  That  should  make  it  easier  to  tailor  the  countermeasures  to  the  circum¬ 
stances  and  so  to  choose  the  set  of  measures  best  suited  to  helping  the  Air  Force  avoid  future 
costs  and  delays  such  as  those  associated  with  the  protests  sustained  in  the  CSAR-X  and  KC-X 
source  selections. 


APPENDIX  A 


A  Simple  Model  of  the  Pattern  of  Protests  and  Protest  Outcomes 


Model  Says  “Corrective  Action  Cost”  Is  Likely  to  Be 
Higher  with  Merit  Review  Than  Without 

'Absent  empirical  data,  an  economic  model  can  offer 
insights  on  relative  costs  of  corrective  action 

*  Model  compares  alternative  assumptions  in  terms  of: 

-  Consistency  of  their  implications  with  observed 
protest  patterns 

-  Implications  for  the  cost  of  corrective  action  with 
and  without  merit  review 

*  Model  predicts  higher  cost  with  merit  review  but... 

*  It  is  impossible  to  say  how  much  higher  without 
knowing  the  inherent  characteristics  of  protests 
being  pursued 

RAND 
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It  is  clear  that  corrective  actions  can  impose  significant  costs  on  the  government,  whether 
they  result  from  voluntary  negotiation  with  a  protester  or  from  a  GAO  protest  sustainment. 
It  is  also  clear  that  merit  reviews  impose  administrative  costs,  delays,  and  distractions  on  the 
government  that  are  not  present  when  voluntary  corrective  action  occurs.  But,  setting  these 
“protest  costs”  aside,  what  are  the  relative  costs  of  the  corrective  actions  themselves  when  they 
are  initiated  in  these  two  different  ways?1  No  reliable  quantitative  data  exist  to  answer  this 
question.  This  appendix  offers  a  simple  economic  model  that  can  help  us  frame  what  is  cur¬ 
rently  known  and  what  this  implies  about  these  relative  costs. 

It  starts  by  considering  a  situation  in  which  the  government  conducts  source  selections, 
and  events  occur  in  these  source  selections  that  create  potential  opportunities  for  protests.  It 
asks,  which  protests  go  forward?  How  does  the  government  react  to  these  protests?  And  how 
does  that  government  reaction  shape  the  patterns  of  protests  and  protest  outcomes  that  we 


1  This  appendix  distinguishes  “protest  costs” — the  administrative  costs,  delay,  and  distraction  associated  with  a  protest — 
from  “corrective  action  costs” — the  costs  (or  benefits)  that  accrue  outside  the  administrative  bounds  of  the  protest  itself. 
This  is  the  same  distinction  used  in  the  text.  We  typically  present  these  phrases  in  quotation  marks  in  this  appendix  to 
emphasize  that  these  are  terms  of  art  that  are  nonstandard  but  have  specific  meanings  in  this  context. 


137 


138  Government  Accountability  Office  Bid  Protests  in  Air  Force  Source  Selections:  Evidence  and  Options 


observe?  It  offers  a  simple  model  based  on  stylized  facts  that  are  consistent  with  the  informa¬ 
tion  we  gathered  in  interviews  in  the  Air  Force.  It  then  asks  if  the  model  can  generate  patterns 
of  protests  and  protest  outcomes  that  are  compatible  with  what  we  have  observed.  Finally,  it 
posits  some  conclusions  based  on  the  model  and  its  compatibility  with  what  we  observe  in  the 
world. 


The  Stylized  Facts  and  Model 

Let: 


CO  =  cost  to  the  protester  of  initiating  a  protest  and  seeing  it  through  to  the  start  of  a 
merit  review 

CP  =  cost  to  the  protester  of  seeing  a  protest  through  a  merit  review 

CG  =  cost  to  the  government  of  seeing  a  protest  through  a  merit  review 

BA  =  benefit  to  the  protester  of  receiving  corrective  action 
CA  =  cost  to  the  government  of  providing  corrective  action 
PC  =  probability  that  the  government  prevails  in  merit  review. 

Assume  the  following  (we  will  adjust  assumptions  later  if  they  prove  to  yield  unrealistic 
results): 


•  Each  opportunity  for  a  protest  is  unique  and  is  accompanied  by  a  set  of  values  for  the  six 
attributes  listed  above.  The  protester  and  government  share  exactly  the  same  beliefs  about 
these  values.  These  beliefs  entail  no  uncertainty. 

•  A  joint  probability  distribution  exists  for  these  six  attributes.  Opportunities  for  protest 
appear  as  random  draws  from  this  distribution.  The  values  associated  with  each  attribute 
in  each  draw  are  fixed  and  beyond  the  control  of  the  protester  or  government. 

•  The  opportunities  for  protests  appear  to  the  protester,  which  decides  whether  or  not  to 
follow  each  opportunity  and  file  a  protest.  When  the  protester  files  a  protest,  the  gov¬ 
ernment  decides  whether  to  offer  voluntary  corrective  action  or  move  directly  to  a  merit 
review.  When  the  government  decides  to  offer  corrective  action,  the  protester  decides 
whether  to  accept  it  or  move  to  merit  review.  If  a  protest  reaches  merit  review,  the  GAO 
review  yields  an  outcome  through  a  random  process  based  on  the  probability  of  govern¬ 
ment  success  associated  with  the  original  protest  opportunity.  Figure  A.l  summarizes  this 
sequence  of  events. 

•  The  cost  of  corrective  action  to  the  government  is  the  same  whether  the  government  vol¬ 
unteers  it  or  GAO  requires  it.  The  benefit  of  corrective  action  to  the  protester  is  the  same 
whether  the  government  volunteers  it  or  GAO  requires  it. 

Start  with  the  final  decision  in  Figure  A.l  and  work  back  to  the  beginning.  When  the 
government  offers  the  protester  a  voluntary  corrective  action, 
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Figure  A.1 

Decisions  Relevant  to  Protest  Patterns  and  Outcomes 


RAND  DB603-A.  1 


If  BA  >  (1  -  PG)*BA-  CP,  the  protester  accepts.  Stated  differently,  this  condition  is  PG*BA 
+  CP  >  0,  which  always  holds.  A  protester  accepts  every  corrective  action  offered  (Link 
5)  and  rejects  none  (Link  6). 

When  the  government  receives  a  protest: 

If  (1  -  PG)*CA  +  CG  >  CA,  it  offers  voluntary  corrective  action.  Equivalently,  CG  > 
PG*CA.  When  this  occurs,  a  protest  moves  through  Link  3. 

If  (1  -  PG)*CA  +  CG  <  CA,  it  goes  directly  to  merit  review.  Equivalently,  CG  <  PG*CA. 
When  this  occurs,  a  protest  moves  through  Link  4. 

Anticipating  this,  the  protester  protests 

If  (1  -  PG)*CA  +  CG  >  CA  and  BA  >  CO  or  if  (1  -  PG)*CA  +  CG  <  CA  and  (1  -  PG)*BA 
>  CO  +  CP.  When  either  combination  of  conditions  applies,  the  potential  protest  moves 
through  Link  1.  All  other  potential  protests  move  through  Link  2,  and  no  protest 
occurs. 

In  sum,  the  following  conditions  exist  along  each  link: 

Link  1:  [CG  >  PG*CA  and  BA  >  CO]  or 

[CG  <  PG*CA  and  (1  -  PG)*BA  >  CO  +  CP] 

Link  2:  Conditions  in  Link  1  do  not  apply. 
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Link  3:  CG  >  PG*CA. 

Link  4:  CG  <  PG*CA. 

Link  5:  BA  >  (1  —  PG)*BA  —  CP.  This  always  holds,  because  PG*BA  +  CP  >  0. 

Link  6:  Empty 

Link  7:  Government  succeeds  in  merit  review,  which  occurs  PG  of  the  rime. 

Link  8:  Government  fails  in  merit  review,  which  occurs  (1  -  PG)  of  the  time. 


The  following  conditions  apply  in  the  outcomes  for  the  four  subsets  into  which  the  model 
sorts  potential  protests: 


Outcome  1: 
Outcome  2: 

Outcome  3: 

Outcome  4: 


Conditions  in  Link  1  do  not  apply.  No  protest  occurs. 

Conditions  in  Links  1  and  3  apply.  Voluntary  corrective  action  occurs; 
merit  review  does  not. 

Conditions  in  Links  1  and  4  apply.  Merit  review  occurs  and,  PG  of  the 
time,  the  government  prevails. 

Conditions  in  Links  1  and  4  apply.  Merit  review  occurs  and,  (1  -  PG)  of 
the  time,  the  government  fails. 


Implications 

Implication  1:  Protesters  always  accept  a  corrective  action  when  it  is  offered.  (Link 
6  is  empty.)  We  do  observe  protesters  rejecting  voluntary  Air  Force  corrective  actions.  So  the 
stylized  facts  miss  something  important.  Consider  two  possibilities. 

Perhaps,  contrary  to  the  assumptions  in  the  model,  the  government  can  affect  the  amount 
of  corrective  action  it  offers  in  a  particular  case  and  recognizes  that  it  need  not  offer  as  much 
corrective  action  as  GAO  might  require  to  induce  a  protester  to  forgo  a  merit  review.  If  the  gov¬ 
ernment  guesses  wrong  about  what  to  offer  in  this  case,  a  protester  may  reject  the  government 
offer.  This  seems  unlikely  if  negotiation  is  feasible  and  the  government  and  protester  agree  on 
the  values  of  all  relevant  attributes. 

Alternatively,  contrary  to  the  assumptions  in  the  model,  a  protester  gains  something 
beyond  the  expectation  of  winning  when  it  moves  to  a  merit  review.  Many  have  argued  that 
protesters  can  delay  a  contract  award  in  this  way  and  either  extend  their  existing  contract  or 
buy  time  to  be  more  competitive  when  a  source  selection  resumes,  making  the  benefit  from 
the  corrective  action  greater  to  them  if  the  source  selection  resumes  later.  This  sounds  more 
plausible.  When  this  is  relevant  to  a  particular  potential  protest,  a  protester  may  be  willing  to 
protest  even  if  it  believes  that  it  has  no  chance  of  winning  in  a  merit  review.  This  might  help 
explain  why  the  government  prevails  so  often  in  merit  reviews. 

Implication  2:  All  else  equal,  the  government’s  willingness  to  offer  corrective  action 
falls  as  the  probability  of  success  in  merit  review  rises.  This  is  true  because  the  govern¬ 
ment  offers  voluntary  corrective  action  when  CG  >  PG*CA.  Therefore,  the  expected  degree  of 
government  success  in  a  protest  should  be  lower  when  the  government  offers  corrective  action 
than  when  it  chooses  to  go  directly  to  merit  review.  Our  interviews  in  the  Air  Force  support 
this  expectation  and  in  fact  identify  it  as  a  reason  to  expect  that  “corrective  action  costs”  are 
probably  about  the  same  whether  the  Air  Force  offers  corrective  action  or  is  forced  to  do  so  fol- 
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lowing  a  merit  review.  In  any  specific  situation,  the  Air  Force  has  a  limited  ability  to  affect  this 
cost  and  must  react  by  deciding  whether  to  offer  corrective  action. 

Implication  3:  All  else  equal,  the  government’s  willingness  to  offer  corrective  action 
falls  as  the  level  of  the  cost  of  corrective  action  rises.  This  is  also  a  result  of  the  fact  that  (in 
the  model)  the  government  offers  voluntary  corrective  action  when  CG  >  PG*CA.  This  may 
be  a  bit  trickier  than  the  implication  above,  because  a  reasonable  set  of  beliefs  would  allow  CG 
and  CA  to  move  together.  The  higher  the  cost  of  a  corrective  action,  the  higher  the  optimal 
amount  the  government  would  likely  expend  to  deal  with  it  if  it  goes  to  a  merit  review.  But  if 
the  government  believes  that  the  fixed  costs  to  going  to  merit  review  are  significant  relative  to 
the  total  costs,  this  implication  would  still  apply.  Another  way  to  say  this  is  that,  the  smaller 
the  expected  “protest  costs”  relative  to  the  expected  “corrective  action  costs,”  the  less  likely  the 
government  is  to  offer  voluntary  corrective  action.  If  the  share  of  “protest  costs”  in  total  costs 
falls  as  total  costs  rise,  the  observed  cost  of  corrective  actions  should,  on  average,  be  higher 
for  actions  that  result  from  merit  review  than  for  those  that  do  not.  This  could  well  result  in 
significantly  higher  observed  costs  for  corrective  actions  resulting  from  merit  reviews  than  for 
those  resulting  from  voluntary  action,  even  if,  in  any  particular  case,  the  costs  do  not  differ. 

Implication  4:  We  cannot  infer  the  relative  sizes  of  “corrective  action  costs”  with  and 
without  merit  review  from  the  government’s  observed  rate  of  success  in  merit  reviews. 
This  success  rate  depends  on  only  cases  that  go  to  merit  review,  and  the  model  makes  it  clear 
that  these  cases  differ  systematically  from  those  that  result  in  voluntary  corrective  actions. 
Figure  A.2  illustrates  this.  It  shows  the  “corrective  action  cost”  (CA)  on  the  abscissa  and  prob¬ 
ability  of  government  success  (PG)  on  the  ordinate.  The  dotted  curve,  CG  =  PG*CA,  traces 
the  boundary  that  divides  cases  where  the  government  offers  voluntary  corrective  action  from 
those  where  it  does  not  if  “protest  cost”  (CG)  is  constant.  If  CG  rises  with  CA,  as  is  likely,  the 
boundary  would  look  more  like  the  solid  curve.  The  shaded  area  posits  a  distribution  of  the 
values  of  CA  and  PG  in  the  subset  of  potential  protests  that  the  protester  has  chosen  to  pursue. 

The  observed  government  success  rate  is  the  average  of  the  values  of  PG  for  the  protests 
in  the  shaded  area  above  the  solid  curve.  The  (unobserved)  average  cost  to  the  government  of  a 
sustained  protest  is  a  weighted  average  of  the  values  of  CA  for  these  same  protests.  The  weight 
for  the  ith  case  in  this  group  is 


n 

O-TT^/JO-PG)), 

t= 1 


where  there  are  n  cases  in  the  group.  This  gives  cases  with  lower  probabilities  of  government 
success — and  so  higher  values  of  CA — greater  weight.  The  (unobserved)  average  cost  to  the 
government  of  a  voluntary  corrective  action  is  simply  the  average  of  CA  for  the  protests  below 
the  solid  line.  By  itself,  given  a  set  of  cases  such  as  those  in  the  shaded  area,  the  observed  gov¬ 
ernment  success  rate  does  not  serve  as  a  useful  summary  statistic  that  can  tell  us  something 
specific  about  these  other  two  values,  even  when  we  know  something  about  the  attributes  of 
the  protests  the  protester  has  chosen  to  pursue. 
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Figure  A. 2 

Comparison  of  Protests  for  Which  the  Government  Offers  Voluntary  Corrective 
Action  and  Those  for  Which  It  Does  Not 


Other  Observations 

There  are  two  reasons  not  yet  discussed  to  raise  questions  about  the  realism  of  this  model. 
First,  our  interviews  in  the  Air  Force  indicated  that,  contrary  to  the  assumption  in  the  model, 
the  government  does  have  some  ability  to  affect  the  cost  of  a  voluntary  corrective  action.  If  we 
allow  that  in  the  model  and  continue  to  assume  that  the  government  and  protester  share  the 
same  beliefs  about  all  attribute  values,  we  have  to  question  why  the  government  would  ever 
offer  more  corrective  action  than  that  required  to  induce  a  protester  to  accept  it  rather  than 
moving  to  merit  review.  In  fact,  both  the  government  and  the  protester  could  avoid  “protest 
costs”  by  settling  before  a  merit  review.  Merit  reviews  would  simply  cease  and  be  replaced  by 
settlements  that  achieved  outcomes  very  similar  to  what  a  merit  review  would  have  yielded. 

If  that  occurred  for  all  protests,  the  costs  of  voluntary  corrections  would  be  about  the 
same  as  those  for  GAO-directed  corrections  when  the  government  had  little  chance  of  win¬ 
ning  and  just  a  fraction  of  GAO-directed  costs  when  the  government  had  a  high  chance  of 
winning.  In  fact,  the  government’s  ability  to  adjust  corrective  actions  is  limited,  preserving  a 
role  for  merit  reviews.  But  even  if  the  government  could  settle  a  significant  portion  of  protests 
through  negotiations  that  reduced  its  costs  of  voluntary  corrective  action,  the  logic  displayed 
in  Figure  A.2  still  applies.  We  could  not  infer  what  the  settlements  would  look  like  in  the  vol¬ 
untary  agreements  by  examining  only  the  government’s  success  rate  in  merit  reviews. 

Second,  the  model  is  as  simple  and  orderly  as  it  is  because  we  have  assumed  that  the 
government  and  protesters  share  the  same  beliefs  about  the  values  of  our  six  core  attributes. 
The  large  number  of  protests  withdrawn  by  protesters  and  dismissed  by  GAO,  even  when  a 
corrective  action  is  not  offered,  points  to  a  high  degree  of  naivete  among  real  protesters.  If, 
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contrary  to  the  assumptions  in  the  model,  many  protesters  were  naive,  it  would  be  inappropri¬ 
ate  to  assume  that  protesters  and  the  government  agree  on  the  values  of  the  six  attributes  of 
any  potential  protest.  Such  disagreement  would  further  complicate  any  effort  to  infer  relative 
values  of  costs  associated  with  corrective  actions  based  on  the  observed  pattern  of  government 
wins  in  merit  reviews. 

That  said,  protesters  are  more  likely  to  share  government  beliefs  in  acquisitions  with  large 
enough  stakes  to  impose  significant  costs  when  corrective  action  occurs.  For  these  protesters, 
something  like  the  model  offered  here  probably  offers  useful  insights.  But  even  in  these  large, 
complex  acquisitions,  beliefs  can  differ.  One  way  to  interpret  the  CSAR-X  and  KC-X  protests 
is  to  suggest  that  the  protesters  involved  discovered  methods  that  the  government  had  not 
anticipated.  As  a  result,  the  government  treated  the  expected  probabilities  and  costs  of  success¬ 
ful  protests  as  smaller  than  the  protesters  did,  with  serious  implications  for  the  outcomes  of 
the  protests  in  these  acquisitions.  Some  observers  believe  that  the  government  has  responded 
to  these  two  protests  by  becoming  more  conservative  and  offering  voluntary  corrective  actions 
more  often.  One  way  to  interpret  this  is  to  say  that  the  government  has  adjusted  its  beliefs  so 
that  they  are  more  in  line  with  those  of  potential  protesters.  And  so,  the  beliefs  of  the  govern¬ 
ment  and  these  protesters  are  better  aligned  than  they  may  have  been  when  the  CSAR-X  and 
KC-X  protests  occurred. 
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WSARA  Could  Clarify  Requirements , 
Improve  Evaluation,  Reduce  Uncertainty 


Goals  of  WSARA 

Implications  for  source  selection 

Rebuild  government  capa¬ 
bilities  in  cost  assessment, 
DT&E,  systems  engineering 

Improve  technical  evaluation, 
allow  use  of  more  sophisticated 
cost  criteria 

Improve  capabilities  for 
performance  assessment, 
root  cause  analysis 

Improve  acquisition  program 
structure,  cost  estimation 

Trade  off  cost,  schedule,  and 
performance  priorities  in 
requirements  development 

Clarify  relative  importance  of 
source  selection  criteria 

Tighten  rules  for  reporting 
cost,  schedule  growth  to 
Congress 

improve  cost  estimates  early  on, 
thereby  improving  source 
selection  outcomes 

Tighten  compliance  with  DAB 
milestone  certifications 

Reduce  uncertainty  about 
technologies 

GAO  Bid  Protests  in  Air  Force  Source  Selections  -57  - 


The  Weapon  Systems  Acquisition  Reform  Act  of  2009  made  a  number  of  important 
changes  in  the  organization  of  DoD  systems  acquisition  and  the  policies  that  govern  it.1  These 
changes  will  help  define  a  new  environment  for  source  selections  and,  hence,  for  protests  of 
government  decisions  made  during  source  selection.  This  appendix  briefly  summarizes  the  ele¬ 
ments  of  WSARA  that  could  affect  source  selection  and  so  the  findings  and  recommendations 
developed  in  this  document. 

WSARA  has  the  following  broad  effects: 

•  reorganizes  cost  estimation,  program  assessment,  and  risk  evaluation  in  ways  that  should 
increase  their  level  of  influence  in  DoD  decisionmaking 

•  increases  the  role  of  the  combatant  commanders  in  the  development  of  requirements  for 
new  acquisition  programs 


1  U.S.  Congress,  2009a;  U.S.  Congress,  Conference  Report  to  Accompany  House  of  Representatives  Bill  S.454 ,  Washington, 
D.C.,  May  20,  2009b. 
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•  requires  consideration  of  trade-offs  among  cost,  schedule,  and  performance  objectives 
as  part  of  the  JCIDS  process  for  developing  requirements  that  will  be  used  in  source 
selections 

•  promotes  greater  use  of  competition  and  prototyping 

•  tightens  oversight  of  unplanned  cost  and  schedule  growth 

•  promotes  better  use  of  workforce  incentives 

•  tightens  oversight  of  earned  value  management 

•  promotes  monitoring  of  the  health  of  the  industrial  base  relevant  to  DoD  systems 
acquisition. 

If  these  changes  have  the  intended  effects,  they  should  all  improve  the  general  environment 
in  which  system  acquisition — and  so  source  selection  in  particular — occurs.  But  a  number 
of  these  changes  could  have  more  specific  effects  on  source  selection  and  so  on  the  issues  dis¬ 
cussed  in  this  document.  The  slide  summarizes  these. 

The  act  accepts  as  valid  a  broad  perception  that  DoD’s  ability  to  estimate  costs,  con¬ 
duct  developmental  testing  and  evaluation  (DT&E),  and  perform  system  engineering  has 
declined  with  recent  secular  reductions  in  the  size  and  seniority  of  the  in-house  acquisition 
workforce  and  increasing  dependence  on  outside  providers  of  these  capabilities.  WSARA  seeks 
to  heighten  the  visibility  and  perceived  importance  of  these  capabilities  inside  the  government 
and  to  sharpen  accountability  for  them.  To  the  extent  this  effort  succeeds,  government  SSETs 
should  become  more  sophisticated,  allowing  better  designed  and  more  thoroughly  executed 
evaluations.  Independent  cost  estimation  should  improve,  allowing  the  use  of  more  sophis¬ 
ticated  methods  and  more  reliable  evaluations  of  the  costs  associated  with  proposals.  Such 
improvements  in  technical  sophistication  and  cost  estimation  capability  are  directly  relevant 
to  concerns  about  the  capabilities  of  the  government  workforce  in  general  and  cost  estimators 
in  particular  addressed  in  the  text. 

The  act  requires  that  the  Secretary  of  Defense  identify  a  “senior  official  in  OSD  as  the 
principal  official  of  DoD  responsible  for  conducting  and  overseeing  performance  assessments 
and  root  cause  analyses  for  major  defense  acquisition  programs.”2  Performance  assessment  will 
track  program  cost,  schedule,  and  performance  relative  to  program  targets  of  a  program  and 
determine  whether  the  program  is  likely  to  meet  DoD  requirements  more  cost-effectively  than 
any  alternative  might.  When  such  assessment  identifies  shortfalls,  root  cause  analysis  will  seek 
the  underlying  cause  or  causes.  Possibilities  include,  among  others3 

(I)  unrealistic  performance  expectations;  (2)  unrealistic  baseline  estimates  for  cost  or  sched¬ 
ule;  (3)  immature  technologies  or  excessive  manufacturing  or  integration  risk;  (4)  unantici¬ 
pated  design,  engineering,  manufacturing,  or  technology  integration  issues  arising  during 
program  performance;  (5)  changes  in  procurement  quantities;  (6)  inadequate  program 
funding  or  funding  instability;  and  (7)  poor  performance  by  government  or  contractor 
personnel  responsible  for  program  management. 

If  these  changes  have  the  desired  effects,  program  discipline  should  improve,  problems 
should  be  caught  earlier,  and  the  Air  Force  and  offerors  should  have  a  better  understanding  of 


2  U.S.  Congress,  2009a,  Sec.  103(a)(1). 

3  U.S.  Congress,  2009a,  Sec.  103(d). 
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the  technological  status  and  likely  future  costs  of  systems  coming  into  source  selection.  This 
should  reduce  uncertainty  and  support  development  of  more  clearly  stated  requirements  and 
requests  for  proposals.  As  noted  in  the  text,  vaguely  crafted  requirements  and  requests  for  pro¬ 
posal  have  induced  protests  in  the  past  and  led  to  sustained  protests  in  the  CSAR-X  and  KC-X 
source  selections. 

The  act  requires  “consideration  of  trade-offs  among  cost,  schedule,  and  performance 
objectives  as  part  of  the  process  for  developing  requirements  for  Department  of  Defense  acqui¬ 
sition  programs.”4  The  JROC  will  ensure  such  consideration  and,  in  particular,  ensure  that 
analyses  of  alternatives  consider  these  trade-offs  for  each  alternative  considered.  That  is,  the 
acquisition  process  will  complete  such  trade-off  assessments  well  before  source  selection  occurs. 
If  these  changes  have  the  effects  desired,  the  requirements  process  should  clarify  the  rela¬ 
tive  importance  of  requirements  identified,  helping  source  selection  teams  frame  and  evaluate 
requests  for  proposal  more  precisely.  More  clearly  defined  requirements  should  help  the  Air 
Force  address  concerns  discussed  in  the  text.  Note  that  the  act  does  not  require  greater  consid¬ 
eration  of  trade-offs  within  a  source  selection,  as  at  least  one  observer  has  suggested.5 

The  act  tightens  the  rules  under  which  major  programs  identify  shortfalls  and  report 
them  to  Congress.  When  a  large  enough  shortfall  occurs  in  cost,  schedule,  or  performance, 
DoD  must  quickly  report  it  to  relevant  congressional  committees,  explain  the  root  cause  and 
road  ahead  to  remove  the  shortfall,  and  justify  the  continuation  of  the  program  and  the  plan 
for  correcting  the  shortfall  or  terminate  the  program.  Greater  technical  maturity  should  clarify 
the  status  of  the  program  and,  as  above,  simplify  design  and  execution  of  a  cost-effective  source 
selection.  Such  improvements  appear  likely  to  help  the  Air  Force  address  some  of  the  difficul¬ 
ties,  discussed  in  the  text,  that  it  has  had  with  faulty  cost  estimates  in  source  selection. 

Tightening  of  rules  related  to  milestone  certification  should  increase  the  technical  matu¬ 
rity  of  programs  reaching  source  selection,  with  similar  benefits  to  the  source  selection  process. 


4  U.S.  Congress,  2009a,  Sec.  201(a). 

5  Burton,  2009,  for  example. 
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